💻 computer science

CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving

CLEAR introduces a scalable closed-loop reinforcement learning framework for end-to-end autonomous driving that trains a residual waypoint policy on top of pretrained Vision-Language-Action models using a heterogeneous simulation pipeline, achieving state-of-the-art performance on challenging benchmarks by overcoming the distribution shift limitations of traditional imitation learning.

Original authors: Yunxiao Shi, Hong Cai, Mohammad Ghavamzadeh, Fatih Porikli

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunxiao Shi, Hong Cai, Mohammad Ghavamzadeh, Fatih Porikli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a self-driving car. For a long time, the best way to do this was Imitation Learning. Think of this like a student driver sitting in the back seat of a car driven by a perfect expert. The student just copies what the expert does. If the expert turns left, the student turns left.

The problem? This only works in a "closed book" test. If the student sees a situation the expert never showed them, or if the student makes a tiny mistake, they get confused. In the real world (or a complex simulation), one small mistake can lead to a crash, and the student doesn't know how to recover because they were only taught to copy, not to think or adapt.

Recently, researchers started using Vision-Language-Action (VLA) models. These are like super-smart driving instructors who can "see" the road, "read" the traffic signs, and "speak" the driving commands all at once. But even these smart instructors were mostly trained using the "copy the expert" method, leaving them vulnerable when things go wrong.

Enter CLEAR.

The Big Idea: The "Residual" Coach

The authors of this paper built a system called CLEAR to fix this. Instead of just copying, they taught the AI to be a coach that corrects a trainee.

The Pretrained "Trainee": First, they take a very smart VLA model (the trainee) and teach it the basics using a huge library of expert driving data. This trainee is good at predicting a general path, like a GPS saying, "Turn left in 500 feet."
The "Coach" (RL): Then, they introduce Reinforcement Learning (RL). Imagine a coach standing next to the trainee. The trainee makes a prediction (the "base path"), and the coach says, "Actually, turn a little bit more to the left to avoid that pothole."
- The AI doesn't relearn how to drive from scratch (which would take too much computer power).
- Instead, it learns a residual policy. This is like learning the difference between a good plan and a perfect plan. It only learns the small "corrections" needed to fix the trainee's mistakes.

The "Heterogeneous Pipeline": Solving the Traffic Jam

Training these AI coaches is incredibly data-hungry. You need to run millions of driving simulations to teach them.

Here's the bottleneck:

The Simulator (The Road): Running a realistic driving simulator (like CARLA) is heavy on computer graphics. It's like running a high-end video game.
The Learner (The Brain): The AI model is also huge and needs powerful graphics cards (GPUs) to think.

If you try to run the simulator and the brain on the same computer, they fight over the graphics cards. It's like trying to run a marathon and a heavy weightlifting competition in the same small gym at the same time; everything slows down or crashes.

CLEAR's Solution: They built a heterogeneous pipeline.

They put the Simulators (the roads) on one set of older, cheaper computers.
They put the AI Brain (the learner) on a separate, super-powerful cluster of computers.
They connected them with a digital tunnel (SSH).

This is like having a fleet of remote race tracks (the simulators) sending data to a central headquarters (the brain). The tracks don't have to be fancy; they just need to run. The brain gets all the data it needs without being slowed down by the graphics rendering. This allowed them to run 64 simulations at the same time and train on 100 million samples.

The Results: From "Failing" to "Winning"

The paper tested this system on some very difficult driving challenges (like the "longest6" and "Bench2Drive" benchmarks).

Before CLEAR: Previous methods, even the smart ones, failed almost 100% of the time on these hard routes. They would crash or get stuck because they couldn't recover from small errors.
With CLEAR: The system learned to correct its own mistakes. It didn't just follow a script; it learned to navigate around obstacles and handle complex traffic.
- On one benchmark, the success rate jumped from 0% (total failure) to 25%.
- On another, it achieved the highest scores ever recorded, beating all previous methods in driving efficiency and safety.

Summary

Think of CLEAR as taking a smart but rigid student driver, giving them a super-smart coach that learns to nudge them in the right direction when they start to drift, and building a massive, distributed training facility so they can practice millions of miles without the computer crashing. The result is a self-driving system that is significantly better at handling the messy, unpredictable reality of the road.

Technical Summary: CLEAR – Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving

Problem Statement

End-to-end autonomous driving (E2E-AD) aims to map raw sensor data directly to driving actions. Recently, Vision-Language-Action (VLA) models have emerged as a promising paradigm, integrating visual perception, language understanding, and action prediction within a single policy. However, existing VLA-based approaches predominantly rely on Imitation Learning (IL). While IL scales well with data, it suffers from a critical distribution shift between open-loop training (learning from logged expert trajectories) and closed-loop inference (where the agent's own actions affect future states). This shift often leads to suboptimal performance and failure in complex, dynamic scenarios.

Furthermore, applying Reinforcement Learning (RL) to vision-based policies at scale faces significant engineering hurdles. RL is data-hungry and requires thousands of parallel simulation environments. In practice, running the simulator (e.g., CARLA) and the VLA learner on the same hardware creates resource contention (specifically GPU memory and rendering instability), limiting the number of parallel environments and hindering effective training.

Methodology: The CLEAR System

The authors propose CLEAR, a system designed to enable large-scale closed-loop RL finetuning for VLA policies. The approach consists of three core components:

1. Two-Stage Training Framework

Stage 1: Open-Loop Pretraining: The system first pretrains a VLA model (using InternVL3-1B as the backbone) on a large dataset of expert trajectories (~3.1 million samples) using imitation learning. This establishes a "waypoint prior," equipping the model with baseline driving capabilities and robust perception.
Stage 2: Closed-Loop RL Finetuning: Instead of training from scratch or outputting low-level controls directly, CLEAR learns a residual waypoint policy.
- The pretrained VLA generates a base path and speed waypoints.
- The RL agent (optimized via Proximal Policy Optimization, PPO) learns to predict residual corrections ( $\Delta y, \Delta v$ ) to these base waypoints.
- The final waypoint command is the sum of the base plan and the residual, clipped to safe bounds.
- A deterministic controller then maps these waypoints to low-level vehicle controls (steering, throttle, brake). Crucially, the controller is treated as part of the environment dynamics, meaning no gradients flow through it during RL updates.

2. Heterogeneous Finetuning Pipeline

To overcome the scaling bottleneck, CLEAR employs a heterogeneous pipeline that decouples the simulation environment from the learning agent:

Separation of Compute: The CARLA simulators are launched on standalone host machines (using older-generation or consumer-grade GPUs like Tesla V100), while the VLA learner runs on high-performance clusters (e.g., NVIDIA H100).
Communication: Secure Shell (SSH) tunnels facilitate communication between the server (simulator) and client (learner).
Impact: This architecture eliminates resource contention, allowing the system to launch 64 parallel CARLA environments simultaneously. This enables training on 100 million samples with large batch sizes (total batch: 16,384; mini-batch: 4,096), which is critical for the convergence of data-hungry RL algorithms.

3. Reward Function

The RL objective maximizes the expected discounted return based on Route Completion (RC). The reward function includes:

A soft penalty factor for deviations.
Hard penalties (terminating the episode) for collisions, red light violations, or stop sign infractions.

Key Contributions

Closed-Loop RL for VLA: CLEAR is the first system to effectively enable large-scale closed-loop RL finetuning for VLA-based E2E-AD, moving beyond the limitations of pure imitation learning.
Residual Waypoint Policy: By learning corrections around a pretrained waypoint prior rather than direct low-level control, the method effectively leverages the knowledge embedded in the pre-trained VLA while improving robustness in closed-loop settings.
Scalable Heterogeneous Infrastructure: The proposed pipeline resolves the GPU resource contention issue, enabling the deployment of a massive number of parallel simulators (64+) and training on 100M samples, which the authors identify as a key factor in achieving state-of-the-art performance.

Experimental Results

CLEAR was evaluated on three challenging benchmarks:

CARLA longest6 v2: This benchmark features long routes with complex, safety-critical scenarios.
- Previous methods (including large-scale IL and prior IL+RL attempts) achieved a 0% Success Rate (SR), failing every route.
- CLEAR (InternVL3-1B) achieved a 25.00% SR and a 47.24% Route Completion (RC), significantly outperforming all baselines.
Bench2Drive: A benchmark with 220 distinct safety-critical scenarios.
- CLEAR achieved the highest Driving Score (86.8), Success Rate (69.5%), and Efficiency among all compared methods, including other VLA and transformer-based approaches.
nuScenes: A real-world dataset.
- CLEAR was evaluated in a zero-shot setting (no training on nuScenes data). It achieved the best average L2 error (0.49m) and competitive collision rates compared to models trained specifically on nuScenes, demonstrating strong generalization.

Significance and Limitations

The paper claims that CLEAR sets a new state-of-the-art for VLA policies in driving simulators, demonstrating that Reinforcement Learning at scale is essential for closing the gap between open-loop training and closed-loop performance. The work highlights that simply scaling up data via imitation learning is insufficient; closed-loop optimization is necessary to handle distribution shifts.

The authors acknowledge several limitations and directions for future work:

Sim2Real Transfer: The current success is in simulation; deploying to the real world requires investigating Sim2Real transfer techniques.
Scaling Limits: The current scaling is limited by the number of parallel CARLA instances. Improving the stability of CARLA's GPU rendering stack could allow for even greater scaling.
Reward Design: The current reward function aligns with the metric's global optimum. Designing effective rewards for metrics that lack a clear global optimum remains an open research question.