⚡ electrical engineering

On-Policy Distillation of Language Models for Autonomous Vehicle Motion Planning

This paper demonstrates that on-policy generalized knowledge distillation effectively transfers motion planning capabilities from a large language model teacher to a significantly smaller student model, outperforming reinforcement learning baselines and achieving near-teacher performance on the nuScenes benchmark for autonomous vehicle deployment.

Original authors: Amirhossein Afsharrad, Amirhesam Abedsoltan, Ahmadreza Moradipari, Sanjay Lall

Published 2026-04-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amirhossein Afsharrad, Amirhesam Abedsoltan, Ahmadreza Moradipari, Sanjay Lall

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a young, eager apprentice (the Student) how to drive a self-driving car. You have a world-class, genius instructor (the Teacher) who knows every rule of the road, can predict exactly where pedestrians will step, and can plot the perfect, smooth path around any obstacle.

The problem? The genius instructor is a giant, heavy supercomputer. It's too big and expensive to fit inside a real car. The apprentice, however, is small, fast, and cheap enough to fit in the car's dashboard.

The goal of this paper is to figure out the best way to transfer the genius instructor's driving skills into the small apprentice so the car can drive safely on its own.

The Two Teaching Methods

The researchers tested two different ways to teach the apprentice.

1. The "Guess and Check" Method (Reinforcement Learning / RL)

Imagine the apprentice tries to drive a route. After they finish, the genius instructor looks at the path and says, "You made a mistake at the second turn. You should have turned left a bit more."

How it works: The instructor only gives feedback on the specific turns the apprentice actually took. It's like a teacher grading a test by only looking at the answers the student wrote down, without showing them the correct answers for the questions they got wrong.
The Flaw: If the apprentice makes a tiny mistake early on (like turning slightly too wide), the next turn becomes harder to judge. Because the instructor only sees the final result of that specific mistake, the apprentice keeps compounding errors. It's like trying to learn to walk by only getting corrected after you've already fallen over.

2. The "Shadowing" Method (On-Policy Distillation / GKD)

This is the method the paper champions. Imagine the apprentice drives the car, but the genius instructor is sitting right next to them, whispering in their ear every single second.

How it works: As the apprentice decides to turn the wheel, the instructor doesn't just say "Good" or "Bad." Instead, the instructor says, "If you turn left, there's a 90% chance that's perfect. If you turn right, there's a 10% chance that's okay, but if you go straight, that's a 0% chance."
The Magic: The apprentice learns not just what to do, but the probability of every possible move. They learn the "shape" of the correct path. Even if the apprentice makes a small mistake, the instructor's detailed map of "what could have happened" helps them correct course immediately, rather than spiraling out of control.

The Experiment: The Race

The researchers set up a race on a famous driving dataset (nuScenes) using a giant "Teacher" model (8 billion brain cells) and a tiny "Student" model (1.7 billion brain cells).

The Teacher: Drove perfectly, hitting the target path with almost zero error.
The RL Student (Guess and Check): Drove okay, but made big mistakes. It was about 50% worse than the teacher. It tended to drift off course, especially on long trips, because its early mistakes piled up.
The GKD Student (Shadowing): Drove almost exactly like the teacher. Even though it was 5 times smaller, it was only about 5% less accurate than the giant teacher. It stayed safe, avoided collisions, and followed the road smoothly.

Why Did the "Shadowing" Win?

Think of driving coordinates (like "turn 3.5 meters left") as a sequence of numbers.

The RL method is like a teacher who only tells you, "The number 3 is good." It doesn't tell you that 3.1 or 3.2 are also good, or that 9 is terrible.
The GKD method is like a teacher who shows you the whole spectrum: "3 is great, 3.1 is great, 3.2 is okay, but 9 is impossible."

Because driving requires precise, continuous numbers, seeing the whole picture of what is possible helps the small student understand the physics of the car much better than just seeing a single "good" answer.

The Bottom Line

This paper proves that you don't need a supercomputer to drive a car. You can take a super-smart AI, shrink it down to fit in a car's dashboard, and teach it to drive almost as well as the giant version—but only if you teach it the right way.

By using the "Shadowing" method (On-Policy Distillation), the small student learns to see the road the way the expert does, making it a safe, practical, and affordable solution for the future of self-driving cars.

1. Problem Statement

The paper addresses the challenge of deploying Large Language Models (LLMs) for autonomous vehicle (AV) motion planning on resource-constrained onboard hardware.

Context: Recent work (e.g., GPT-Driver) has shown that reformulating motion planning as a language generation problem—where driving scenes are converted to text prompts and trajectories are generated as structured text with chain-of-thought reasoning—yields interpretable and high-performing planners.
The Bottleneck: State-of-the-art LLMs are too large for real-time inference in automotive systems.
The Distillation Challenge: Standard knowledge distillation (training a small student on teacher-generated or ground-truth data) suffers from train-inference distribution mismatch. In autoregressive models, errors in early generated tokens (e.g., the first coordinate of a trajectory) compound over the sequence, causing the student's distribution at inference time to diverge significantly from the training distribution. This is critical in motion planning, where a small coordinate error can lead to unsafe trajectories or collisions.

2. Methodology

The authors propose On-Policy Generalized Knowledge Distillation (GKD) as a solution and compare it against a Dense-Feedback Reinforcement Learning (RL) baseline.

A. Task Formulation

Input: Natural language prompts encoding perception data (object classes, positions, predicted future states) and ego-vehicle states (velocity, heading, history).
Output: A structured text response containing:
1. Notable Objects: Critical entities.
2. Potential Effects: How objects influence the ego vehicle.
3. Meta Action: High-level decision (e.g., "TURN RIGHT").
4. Trajectory: Six waypoints $(x, y)$ over a 3-second horizon.
Models:
- Teacher: Qwen3-8B, fine-tuned via Supervised Fine-Tuning (SFT) on the nuScenes GPT-Driver dataset.
- Student: Qwen3-1.7B (5 $\times$ smaller).

B. Proposed Method: On-Policy GKD

Instead of training on static datasets, GKD trains the student on its own self-generated sequences (on-policy rollouts).

Objective: The student minimizes the divergence between its token-level probability distribution and the teacher's distribution along the trajectory the student actually generated.
Divergence Metric: The authors use the Generalized Jensen-Shannon Divergence (JSD) with $\beta=0.5$ . This balances Forward KL (mode-covering, preventing the student from ignoring plausible tokens) and Reverse KL (mode-seeking, focusing on high-probability tokens).
Training Mechanism: Gradients are computed based on the token-level divergence but stop-gradient is applied to the sampling process. The student treats its generated prefixes as constants, optimizing only the token-level distribution match. This provides dense, full-vocabulary supervision at every step.

C. Baseline: Dense-Feedback RL

The authors compare GKD against a policy gradient approach using the teacher's log-probabilities as a reward signal.

Mechanism: The student generates a rollout. For each sampled token, the advantage is calculated as the difference between the teacher's log-prob and the student's log-prob.
Limitation: This provides only a scalar reward for the sampled token, discarding information about alternative plausible tokens (the full distribution). It is analogous to sparse outcome rewards in RL, whereas GKD offers dense process rewards.

3. Key Contributions

Effective Compression: Demonstrated that on-policy distillation can compress an 8B parameter LLM planner into a 1.7B parameter model (5 $\times$ reduction) while retaining near-teacher performance.
Principled Comparison: Provided a controlled comparison between full-distribution matching (GKD) and sampled-token reward shaping (RL) using identical teacher-student pairs and datasets, isolating the algorithmic impact.
Safety and Accuracy: Showed that GKD significantly outperforms RL in both trajectory accuracy and collision avoidance, highlighting the importance of full-distribution supervision for sequential decision-making tasks.

4. Experimental Results

Experiments were conducted on the nuScenes validation set (5,119 frames).

Metric	Teacher (8B)	GKD Student (1.7B)	RL Student (1.7B)
Avg L2 Error (STP-3)	0.355 m	0.373 m (5% gap)	0.579 m (55% worse than GKD)
Avg L2 Error (UniAD)	0.730 m	0.772 m (6% gap)	1.092 m (41% worse than GKD)
Avg Collision Rate	0.101%	0.138%	0.363% (2.6 $\times$ higher than GKD)
Format Errors	~0.08%	0%	0%

Trajectory Accuracy: The GKD student closely tracks the teacher, with errors compounding much less than the RL baseline as the time horizon increases (1s to 3s).
Safety: The GKD student achieves a collision rate very close to the teacher, whereas the RL baseline has a significantly higher collision rate, indicating that scalar rewards fail to capture the safety constraints encoded in the teacher's full distribution.
Qualitative Analysis: In complex scenarios (e.g., sharp turns), the GKD student produces trajectories much closer to the ground truth than the RL student, which often drifts significantly.

5. Significance and Conclusion

Practical Deployment: The study proves that on-policy distillation is a viable path for deploying LLM-based planners in embedded automotive systems, overcoming the computational constraints of large models without sacrificing safety or performance.
Algorithmic Insight: The results strongly suggest that for sequential generation tasks involving physical constraints (like vehicle dynamics), full-distribution supervision (GKD) is superior to scalar reward shaping (RL). The teacher's probability distribution encodes implicit knowledge about physically plausible continuations (e.g., valid coordinate sequences) that scalar rewards cannot convey.
Stability: GKD demonstrated more stable training dynamics compared to the RL baseline, which showed signs of overfitting and performance degradation after the first epoch.

In summary, the paper establishes On-Policy GKD as the preferred method for distilling LLM-based motion planners, offering a robust solution to the train-inference mismatch problem while enabling the deployment of high-capability, safety-critical AI on resource-limited hardware.