⚡ electrical engineering

ManeuverGPT Agentic Control for Safe Autonomous Stunt Maneuvers

This paper introduces ManeuverGPT, a novel framework that leverages a large language model-based agentic architecture to generate and safely execute high-dynamic autonomous stunt maneuvers, such as J-turns, through iterative prompt refinement and physics-based validation without retraining model weights.

Original authors: Shawn Azdam, Pranav Doma, Aliasghar Moj Arab

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shawn Azdam, Pranav Doma, Aliasghar Moj Arab

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brand-new, incredibly smart robot driver how to perform a J-turn—that dramatic, 180-degree spin you see in action movies where a car reverses, whips around, and speeds off in the opposite direction.

Usually, teaching a robot to do this is like trying to teach a human to ride a unicycle by writing a 5,000-page manual on physics. You have to program every single rule, and if you want the robot to drive a different car (like a heavy truck instead of a sports car), you have to rewrite the whole manual from scratch.

ManeuverGPT is a new, smarter way to do this. Instead of writing a manual, the researchers gave the robot a super-smart "co-pilot" brain (based on Large Language Models, the same tech behind chatbots) and asked it to figure out the moves through trial and error, just like a human learning a new skill.

Here is how it works, broken down into simple concepts:

1. The Three-Person Pit Crew

Instead of one robot trying to do everything, ManeuverGPT uses a team of three specialized "agents" (AI assistants) working together like a pit crew at a race track:

The Translator (Query Enricher): You tell the system, "Do a J-turn!" The Translator takes that vague command and adds all the necessary details, like "Remember, we are in the rain, and this car is slippery." It turns a simple wish into a detailed instruction manual.
The Driver (Driver Agent): This is the one who actually picks the controls. Based on the Translator's notes, it says, "Okay, I'll press the gas to 50%, turn the wheel left, then brake hard." It guesses the numbers.
The Safety Inspector (Parameter Validator): This is the strict coach. Before the car moves, the Safety Inspector checks the Driver's plan. "Whoa, turning the wheel that hard in this car will make it flip! Let's dial it back." If the plan is safe, it gives the green light. If not, it sends the plan back to the Driver to try again.

2. Learning by "Talking" (Not by Rewiring)

The coolest part is how the system learns.

Old Way: To teach a robot a new trick, you usually have to "retrain" it, which is like forcing a human to go back to school for a year to learn a new math formula.
ManeuverGPT Way: It learns by chatting.
1. The team tries a J-turn.
2. The car spins a little too wide.
3. The system looks at the mistake and tells the "Driver Agent": "Hey, you turned too sharply in the second step. Next time, try turning 10% less."
4. The Driver Agent updates its instructions and tries again.

It's like a human learning to drive a manual transmission car. You don't need to rebuild your brain; you just get feedback ("You stalled the engine!") and adjust your foot pressure next time. The system does this over and over until the J-turn is perfect, without ever changing its underlying code.

3. The Results: Sedan vs. Sports Car

The researchers tested this on two very different cars in a video game simulation (CARLA):

The Sedan: A normal, stable family car. The system learned to do the J-turn perfectly very quickly.
The Sports Coupe: A fast, twitchy, high-performance car that is much harder to control.

The Result: The system succeeded 90% of the time with the Sedan and 70% of the time with the Sports Coupe.

Why the difference? The Sports Coupe is like a racehorse—very sensitive. A tiny mistake in the steering makes it spin out. The Sedan is like a calm horse; it forgives small mistakes. The AI had to be much more precise with the Sports Coupe, but it still managed to pull it off without crashing.

4. Why This Matters

Think of autonomous vehicles (self-driving cars) today. They are great at driving to the grocery store, but if a deer jumps out in front of them, they might just slam on the brakes. They aren't built to do "stunt driver" moves to dodge danger.

ManeuverGPT shows that we can teach self-driving cars to be agile and brave without needing to reprogram them from scratch every time we get a new car model. It combines the creativity and reasoning of a human (using language) with the strict safety checks of a computer.

In a nutshell:
This paper proves that if you give a self-driving car a team of AI assistants that can talk to each other, critique each other's plans, and learn from their mistakes through conversation, that car can learn to perform dangerous, high-speed stunts safely—just by being told what to do and getting a little feedback when it messes up.

1. Problem Statement

Autonomous vehicles (AVs) currently struggle to execute high-dynamic, aggressive maneuvers (such as J-turns) required for emergency hazard avoidance. Traditional approaches face significant limitations:

Reinforcement Learning (RL): Requires extensive training data, environment interactions, and often lacks formal safety guarantees.
Model Predictive Control (MPC): Relies on precise dynamic models that may fail to capture extreme driving behaviors at the limits of handling.
Adaptability: Adapting controllers to new vehicle dynamics usually demands significant retraining or model redesign.

The core challenge is to develop a control framework capable of generating and executing complex stunt maneuvers (specifically 180° J-turns) without retraining model weights, while ensuring safety and adapting to different vehicle dynamics through textual prompts alone.

2. Methodology: ManeuverGPT Framework

The authors propose ManeuverGPT, a novel agentic architecture that utilizes Large Language Models (LLMs) as controllers. Instead of learning weights, the system iteratively refines control parameters through a closed-loop, prompt-based process.

A. Agentic Architecture

The system comprises three specialized LLM-driven agents coordinated by an orchestrator:

Query Enricher Agent ( $A_E$ ): Contextualizes user commands. It augments vague instructions with domain-specific details, historical data, and environmental constraints to create an actionable query.
Driver Agent ( $A_D$ ): Generates a structured sequence of maneuver parameters ( $P$ ) based on the enriched query. These parameters include throttle ( $\theta$ ), steering ( $\phi$ ), brake ( $\beta$ ), and reverse status ( $r$ ) for specific phases of the maneuver.
Parameter Validator Agent ( $A_V$ ): Acts as a safety guardrail. It evaluates candidate parameters against a set of constraints ( $C$ $C$ ), including:
- Safety Constraints ( $C_s$ ): Ensuring no collisions occur.
- Operational Limits ( $C_o$ ): Ensuring parameters stay within physical limits (e.g., steering range, throttle bounds).
- If parameters are invalid, $A_V$ rejects them or triggers refinement.

B. Iterative Optimization Loop

The system operates in discrete iterations ( $k$ ):

Generation: $A_D$ generates parameters $P_k$ based on the enriched query $E(Q_k)$ .
Validation: $A_V$ filters $P_k$ to produce feasible parameters $\tilde{P}_k$ .
Simulation: The CARLA simulator executes the maneuver using $\tilde{P}_k$ .
Feedback & Refinement: Performance metrics (heading error, collisions, jerk) are calculated. A cost function $L(\tilde{P}_k)$ is computed. If the cost exceeds a threshold, feedback is generated to refine the prompt for the next iteration ( $Q_{k+1}$ ), guiding the agents to adjust specific parameters (e.g., "reduce steering angle by 5%").

Theoretical Guarantee: The paper presents a theorem proving Finite-Time Feasibility. Under the assumption that the validator keeps parameters in a safe set and feedback reduces the cost function by a constant $\delta$ , the system is guaranteed to find a feasible solution within a finite number of iterations.

3. Key Contributions

Zero-Retrain Framework: A multi-agent LLM framework that generates stunt maneuvers starting tabula rasa (from scratch) without modifying model weights or requiring extensive training data.
Cross-Vehicle Generalization: Demonstrated the ability to adapt to different vehicle dynamics (sedan vs. sports coupe) solely by adjusting textual prompts, eliminating the need for vehicle-specific retraining.
Performance Analysis: Established a rigorous evaluation using established success metrics (final orientation error, collision checks, time constraints) and compared multi-agent vs. single-agent architectures.

4. Experimental Results

Experiments were conducted in the CARLA simulator (v0.9.14) using a J-turn maneuver (a rapid 180° rotation).

Overall Success Rate: The system achieved an 86% success rate across 100 runs.
- Early trials (first 60): 83.3% success.
- Later trials (last 40): Improved to 90.0% success, demonstrating the efficacy of iterative prompt refinement.
Multi-Agent vs. Single-Agent:
- The multi-agent system maintained angle errors below 7° in 76% of trials, compared to only 52% for a single-agent system.
- Optimal performance (<3° error) was achieved in 31% of multi-agent trials vs. 18% for single-agent.
- The Validator Agent was crucial in preventing "drift" in parameter selection (e.g., preventing steering angles >1.0 that cause immediate collisions).
Vehicle Adaptability:
- Sedan: Achieved a 90.11% success rate with a mean angle error of 9.09°. Its forgiving dynamics allowed for greater error margins.
- Sports Coupe: Achieved a 70.57% success rate with a mean angle error of 21.39°. The vehicle's high responsiveness and rear-biased weight distribution amplified small control errors, making it significantly harder to control.
Convergence: Performance improved over 5 batches (epochs), with mean angle error converging toward the optimal 3° threshold.

5. Significance and Future Work

Significance:

Rapid Prototyping: ManeuverGPT offers a flexible approach for developing high-agility safety features without the computational cost of retraining RL models.
Hybrid Control: It validates the potential of combining LLM-based reasoning (for high-level planning and adaptability) with algorithmic validation (for safety and physics constraints).
Safety Envelope Expansion: It suggests a pathway for AVs to perform "accident-free" evasive maneuvers that mimic professional stunt drivers.

Limitations & Future Directions:

Numeric Precision: LLMs struggle with precise floating-point control, necessitating hybrid approaches for fine-grained adjustments.
Safety Guarantees: The current approach lacks the formal mathematical safety guarantees provided by traditional MPC.
Simulation-to-Reality: Bridging the gap between CARLA simulations and real-world deployment remains a significant hurdle.
Future Work: Integrating explicit safety constraints directly into prompts, developing automated re-prompting based on real-time sensor data, and combining language-based planning with specialized reactive control models.