💻 computer science

Bridging Large-Model Reasoning and Real-Time Control via Agentic Fast-Slow Planning

This paper proposes Agentic Fast-Slow Planning, a hierarchical framework that bridges large-model reasoning and real-time control by decoupling perception, symbolic decision-making, and trajectory generation across natural timescales to significantly improve autonomous driving robustness and efficiency.

Original authors: Jiayi Chen, Shuai Wang, Guangxu Zhu, Chengzhong Xu

Published 2026-04-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiayi Chen, Shuai Wang, Guangxu Zhu, Chengzhong Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a self-driving car how to navigate a busy city. You have two very different tools at your disposal:

The "Super-Brain" (Large Language Model): It's incredibly smart, understands complex stories, and can figure out why you want to go somewhere. But it's slow, like a professor thinking deeply about a philosophy question. It's also a bit clumsy with numbers and can't react fast enough to avoid a sudden pothole.
The "Reflex" (Model Predictive Control): It's a race car driver. It reacts instantly, calculates physics perfectly, and keeps the car on the road. But it's "dumb" in a human sense; it doesn't understand the story of the drive. It just follows the map and doesn't know that a red light means "stop" or that a construction zone requires a detour.

The Problem:
Current self-driving cars try to use one or the other, or they mash them together poorly.

If you let the Super-Brain drive, it might take too long to think, causing the car to stall or crash because it can't react fast enough.
If you let the Reflex drive, it might follow the map perfectly but miss a crucial turn because it doesn't "get" the traffic situation.

The Solution: "Agentic Fast-Slow Planning"
The authors of this paper built a new system that acts like a perfect team of a General and a Sergeant.

The Team Structure

1. The General (The "Slow" Layer)

Role: Strategic thinking.
Analogy: Imagine a General sitting in a command center (the Cloud). He looks at a simplified map of the battlefield. He doesn't need to see every single leaf on a tree; he just needs to know, "There's a tank here, and we need to go left."
How it works:
- The car's camera (on the "Edge") takes a picture and quickly turns it into a simple sketch (a topology graph). It's like turning a high-definition photo into a stick-figure drawing. This saves data and bandwidth.
- This sketch is sent to the Cloud General (the LLM). The General thinks, "Okay, the road is blocked, so the plan is: Go Left, then Keep Straight, then Turn Right."
- He sends these symbolic orders back to the car.

2. The Sergeant (The "Fast" Layer)

Role: Tactical execution.
Analogy: The Sergeant is on the ground, driving the tank. He doesn't need to know why he's turning left; he just needs to know how to turn left safely without hitting a wall.
How it works:
- The car receives the General's orders ("Go Left").
- It uses a smart search algorithm (Semantic-Guided A*) to turn those words into a physical path. It's like a GPS that knows, "The General said 'Go Left,' so I will prioritize paths that go left, but I'll still avoid the wall."
- The "Agent" Twist: Sometimes the Sergeant gets stuck or the map is weird. The system has a "Self-Correcting Agent." If the path looks shaky, this agent acts like a mechanic who tweaks the engine settings on the fly, learning from past mistakes so it doesn't need a human to fix it later.

3. The Reflex (The Control Layer)

Role: Muscle memory.
Analogy: This is the actual steering wheel and gas pedal. Once the Sergeant has drawn the perfect line on the ground, the Reflex layer ensures the car stays exactly on that line, adjusting for wind, bumps, or slippery roads in milliseconds.

Why is this better?

The paper tested this system in a video game simulator called CARLA (which is like a realistic driving video game) and compared it to other methods.

Old Way (Just Reflex): The car gets confused by complex traffic, swerves wildly, and takes a long time to get through.
Old Way (Just Super-Brain): The car crashes because it's thinking too hard about the philosophy of driving while a truck is coming at it.
New Way (The Team):
- Speed: The car finished the course 12% faster.
- Stability: The car stayed much straighter, reducing "wobbles" (lateral deviation) by 45%.
- Safety: It understood the intent of the drive (e.g., "avoid the construction") while still reacting instantly to the physics of the road.

The Big Picture

This paper is about decoupling the job.

Let the Cloud do the heavy, slow thinking (the "Why").
Let the Car do the fast, precise math (the "How").
Connect them with a simple language (stick-figure maps and clear orders) so they don't get confused.

It's like having a brilliant navigator in the passenger seat shouting clear, simple directions, while a professional driver focuses entirely on steering the wheel. The result is a drive that is both smart and safe.

1. Problem Statement

The integration of Large Foundation Models (LFMs) into autonomous driving systems faces a critical challenge: mapping high-level semantic intent to reliable, real-time control. Existing approaches suffer from two main limitations:

Direct LLM Trajectory Generation: While flexible, generating trajectories directly from LLMs is brittle, difficult to verify, and often violates strict real-time latency constraints.
LLM-Adjusted MPC: Adjusting Model Predictive Control (MPC) objectives via language mixes slow deliberation with fast control, blurring the interface between reasoning and actuation and often failing to leverage the complementary strengths of both paradigms.

Current systems fail to decouple perception, reasoning, planning, and control across their natural timescales, leading to inefficiencies in edge-cloud communication and a lack of robustness in dynamic environments.

2. Methodology: Agentic Fast–Slow Planning (AFSP)

The authors propose Agentic Fast–Slow Planning (AFSP), a hierarchical framework inspired by dual-process cognition. It decouples the pipeline into three layers operating at different rates:

Reasoning Layer (Slow): Interprets the scene and issues symbolic decisions.
Planning Layer (Mid-speed): Converts decisions into executable trajectories.
Control Layer (Fast): Ensures real-time feasibility and safety.

The framework is built around two key "bridges":

A. Perception2Decision (Edge-Cloud Bridge)

This module addresses the bandwidth and latency issues of transmitting raw images to the cloud.

On-Vehicle (Edge): A lightweight Vision-Language Model (VLM) Topology Detector processes raw camera images. Instead of sending images, it compresses the scene into a compact ego-centric topology graph ( $Z$ ) containing object semantics, bounding boxes, distances, and orientations.
Cloud: A large LLM Decision Maker consumes the topology graph. Using Chain-of-Thought (CoT) prompting, it generates a sequence of symbolic driving directives (e.g., LEFT, KEEP, RIGHT) and a driving style (reference velocity).
Benefit: This reduces bandwidth significantly while preserving interpretability and shifting heavy reasoning to the cloud.

B. Decision2Trajectory (Reasoning-Planning Bridge)

This module translates symbolic directives into executable paths, addressing the brittleness of classical planners.

Semantic-Guided A*: Instead of treating LLM directives as hard constraints (which can cause deadlocks), the algorithm embeds them as soft costs in the A* search function.
- The state space is augmented with the current directive index and previous move.
- Transitions are categorized as Correct, Delay, Wrong, or Overact, assigning specific semantic costs ( $c_{sem}$ ).
- This biases the search toward intent-consistent paths while allowing recovery if geometry conflicts with the directive.
Agentic Refinement Module: To handle the sensitivity of A* hyperparameters to scene variations, an LLM-based agent acts as a closed-loop tuner.
- It retrieves warm-start parameters from a cloud memory based on the current scene.
- It iteratively refines hyperparameters (e.g., cost weights) based on trajectory feedback (metrics like oscillation or deviation) until a valid path is found.
- This automates the "trial-and-error" process usually requiring human expertise.

C. Cloud-Guided MPC

The final trajectory is tracked by an MPC controller. The system uses a switching mechanism:

Nominal Conditions: Uses a local lightweight reference for low-latency control.
Difficult Cases: Switches to a cloud-provided global reference if the local planner struggles, ensuring safety without sacrificing responsiveness.

3. Key Contributions

Hierarchical Framework (AFSP): A novel architecture that explicitly decouples perception, reasoning, planning, and control across timescales, preventing the "blurring" of interfaces seen in previous hybrid approaches.
Perception2Decision: A mechanism that compresses visual data into topological graphs on the edge and offloads semantic reasoning to the cloud, achieving high decision quality with minimal latency and bandwidth.
Decision2Trajectory:
- Semantic-Guided A*: The first principled extension of A* that incorporates language-derived semantics as soft costs, ensuring intent consistency without sacrificing geometric feasibility.
- Agentic Refinement: An automated, memory-augmented loop that tunes planner hyperparameters online, eliminating the need for manual retuning.
Empirical Validation: Comprehensive experiments demonstrating significant improvements in robustness and efficiency over state-of-the-art baselines.

4. Experimental Results

Experiments were conducted in the CARLA simulator across three scenarios (nominal map, shifted map with enlarged obstacles, and complex perturbations). The proposed AFSP was compared against pure MPC and an A-guided MPC* baseline.

Robustness: AFSP significantly reduced lateral deviation by up to 45% compared to pure MPC and 35% compared to A*-MPC.
Efficiency: The system reduced completion time by over 12% compared to MPC and 11% compared to A*-MPC.
Smoothness: AFSP produced trajectories with lower speed variance and fewer oscillations, particularly in perturbed environments where classical planners failed to find optimal corridors.
Perception/Decision Efficiency: The VLM-to-LLM pipeline achieved comparable decision quality to direct VLM inference but with significantly lower latency (4.13s vs. 10.24s for VLM-only in specific tests) and much lower data transmission requirements.

5. Significance

This work represents a significant step toward practical, safe, and interpretable autonomous driving using foundation models.

Bridging the Gap: It successfully bridges the gap between the "slow" reasoning capabilities of LLMs and the "fast" requirements of real-time control, avoiding the pitfalls of end-to-end learning or naive parameter tuning.
Scalability: By utilizing an edge-cloud architecture with topology compression, it offers a scalable solution for deploying large models in resource-constrained vehicles.
Safety & Interpretability: The use of symbolic directives and soft-cost constraints ensures that the system's reasoning is transparent and that the resulting trajectories are geometrically feasible and safe, addressing the "black box" concerns of pure learning-based approaches.

The code is available at https://github.com/cjychenjiayi/icra2026_AFSP.