⚡ electrical engineering

Knowledge-Data-Dual-Driven Reinforcement Learning for Autonomous Vehicle Control in Mixed Traffic

This paper proposes Knowledge-Data Dual-driven Reinforcement Learning (KDDRL), a framework that synthesizes intention-aware trajectories and fuses probabilistic data with physical constraints to enable asynchronous multi-timescale optimization, thereby enhancing the safety, efficiency, and comfort of autonomous vehicle control in mixed traffic.

Original authors: Jie Fang, Wei Zheng, Mengyun Xu, Eui-Jin Kim

Published 2026-08-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jie Fang, Wei Zheng, Mengyun Xu, Eui-Jin Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to drive a car through a busy city. This isn't just about following a map; it's about playing a high-speed game of chess where the other pieces are real people who might suddenly swerve, brake, or change their minds. This is the world of Autonomous Vehicles (AVs), a field where computer scientists try to build cars that can think and react on their own. To do this, they often use a technique called Reinforcement Learning (RL). Think of RL like training a dog: the computer "dog" tries different actions, gets a "treat" (a reward) for good behavior like staying in the lane, and a "scolding" (a penalty) for bad behavior like crashing. Over time, the dog learns the best tricks to get the most treats. However, there's a catch: real drivers are messy and unpredictable. They don't always follow the rules, and their intentions are hidden. If the robot car only looks at what is happening right now, it might be too slow to react when a human driver suddenly cuts them off. This paper tackles the problem of teaching these robot cars to be smarter, safer, and more comfortable drivers when sharing the road with humans.

The researchers behind this study, Jie Fang and their team, realized that current robot drivers have three big headaches. First, they are often too passive; they react to what they see but can't guess what a human driver intends to do next. Second, they struggle with "long-tail" events—those rare, scary moments like a sudden cut-in that happen so rarely the robot never learns how to handle them. Third, driving involves two very different types of actions happening at different speeds: steering and changing lanes (which happens rarely and is a big decision) versus speeding up or slowing down (which happens constantly and needs to be smooth). Trying to teach a robot to do both at the exact same time often confuses the learning process.

To solve this, the team built a new system called Knowledge-Data Dual-driven Reinforcement Learning (KDDRL). You can think of KDDRL as giving the robot car a "crystal ball" and a "rulebook" at the same time. Instead of just looking at the car in front of it, the robot uses a special AI tool (a type of generative model) to imagine several possible futures. It asks, "If that car keeps going straight, where will it be? What if it gets scared and swerves?" This turns the robot from a passive observer into a proactive predictor.

But guessing isn't enough; the robot also needs to be safe. So, the system mixes these guesses with hard physical rules (like "don't hit the car in front") and real-world data about how humans actually drive. It's like having a wise driving instructor (the physics rules) sitting next to a student who has read a million driving stories (the data). The instructor makes sure the student doesn't try anything dangerous, while the stories help the student understand that sometimes people act weirdly.

The paper also fixes the "two-speed" problem. The robot uses two different brains working together. One brain, running very fast (every 0.1 seconds), handles the smooth gas and brake pedals for following cars. The other brain, running slower (every 2 seconds), handles the big decisions like changing lanes. These two brains talk to each other and share a "common language" of what's happening, so the car doesn't get confused.

The researchers tested this system in a computer simulation that used real driving data from German highways to make the traffic look realistic. They found that their new method, KDDRL, helped the robot car learn faster and drive more safely than older methods. In these simulations, the robot was better at guessing what other drivers would do, handled rare scary situations more effectively, and kept the ride smooth and comfortable. The authors suggest that by combining the ability to predict human intentions with strict safety rules, we can build autonomous cars that are ready for the messy, unpredictable reality of mixed traffic.

Technical Summary: Knowledge-Data-Dual-Driven Reinforcement Learning for Autonomous Vehicle Control in Mixed Traffic

Problem Statement
The paper addresses three interrelated challenges in autonomous vehicle (AV) decision-making within mixed traffic environments (interactions between AVs and human-driven vehicles, HDVs):

Limitations of Physics-Based Priors: Existing physics-informed reinforcement learning (RL) models rely on deterministic physical models (e.g., kinematic equations, MOBIL) that fail to capture latent interactive intentions and diverse driver behaviors. This limits the agent's ability to proactively reason about future vehicle evolution.
Non-Stationarity and Long-Tail Events: Safety-critical scenarios involving abrupt maneuvers by surrounding vehicles cause distributional shifts. Standard RL struggles with these non-stationary conditions, leaving rare but critical safety events (long-tail events) under-explored.
Hybrid Action Space Instability: AV control requires managing continuous car-following (longitudinal) and discrete lane-changing (lateral) maneuvers. These operate at fundamentally different temporal scales. Forcing them into synchronous updates destabilizes training, while simple decoupling isolates interdependent decision contexts.

Methodology: KDDRL Framework
The authors propose Knowledge-Data Dual-driven Reinforcement Learning (KDDRL), a framework that shifts from passive perception-reaction to proactive prediction-reasoning. The architecture consists of three synergistic components:

Intention-Aware Trajectory Generation Submodule:
- Instead of using raw observations, the system generates future trajectories of leading vehicles that encode interactive intentions.
- It employs a Conditional Generative Adversarial Network (cGAN), termed trajGAN.
- Process: An Intelligent Driver Model (IDM) first generates a deterministic physics-based prior trajectory. This serves as a conditioning sequence ( $T_{CT}$ ) for the trajGAN, which refines it into multiple physically plausible, intention-aware trajectories ( $T_{GT}$ ) by learning from real-world data distributions.
- Output: Key relative distances at specific future time steps (10th and 20th steps) are extracted to provide structured foresight without exploding state dimensionality.
Lane-Changing Behavior Generation Submodule:
- This module fuses data-driven insights with physical constraints to guide lane-changing decisions.
- Data-Driven Component: A lcGAN learns empirical lane-changing patterns from real-world datasets, outputting a probability distribution ( $P_{lcGAN}$ ) conditioned on traffic states and the intention-aware trajectories. It utilizes a Focal loss to address class imbalance between frequent lane-keeping and rare lane-changing maneuvers.
- Knowledge-Driven Component: The MOBIL model provides a physics-based prior ( $P_{MOBIL}$ ) by enforcing hard safety distance constraints and calculating utility based on efficiency and safety criteria.
- Fusion: An embedding layer compresses $P_{lcGAN}$ and $P_{MOBIL}$ into a compact shared prior distribution ( $P_{lc}$ ), which serves as an inductive bias for the RL agent.
Vehicle Control Module (Hierarchical RL):
- The framework employs a unified predictive-knowledge state basis that integrates raw observations, intention-aware trajectories, and the fused lane-changing prior.
- Asynchronous Multi-Timescale Optimization:
  - Discrete Lane-Changing: Handled by a D3QN (Dueling Double Deep Q-Network) agent operating at a low frequency (2-second intervals). It selects actions (Left, Keep, Right) based on the unified state.
  - Continuous Car-Following: Handled by a TD3 (Twin Delayed Deep Deterministic Policy Gradient) agent operating at a high frequency (0.1-second intervals). It outputs continuous acceleration commands.
- The two agents share the same state representation but update asynchronously, preserving mutual information while stabilizing training for heterogeneous action spaces.

Key Contributions

Improved Intention-Aware Trajectory Generation: The paper introduces a cGAN-based model that encodes interactive intentions to synthesize multiple physically plausible future trajectories, converting passive perception into proactive predictive states.
Knowledge-Data Dual-Driven Paradigm: A novel training approach that fuses probabilistic data-driven insights (from lcGAN) with interpretable physical boundaries (from MOBIL) on predictive states. This stabilizes policy optimization in non-stationary regimes and improves exploration efficiency in rare safety-critical events.
Shared Predictive-Knowledge State Basis: A coupling module that compresses intention-aware trajectories and physical constraints into shared embeddings. This enables asynchronous optimization of continuous and discrete actions, resolving the instability of hybrid action spaces while maintaining coordinated control.

Results
Evaluations were conducted in a simulation environment calibrated with the HighD dataset (real-world German highway trajectories) integrated into the SUMO traffic simulator. The KDDRL framework was compared against conventional baseline methods. The results demonstrate that KDDRL:

Effectively handles intention uncertainty.
Accelerates training convergence.
Outperforms baselines in terms of safety (reduced collisions), efficiency (speed maintenance), and comfort (reduced jerk and unnecessary maneuvers).

Significance and Claims
The paper claims that KDDRL represents a significant advancement by addressing the structural coupling of prediction, safety constraints, and control frequency in mixed traffic. By shifting the paradigm from "passive perception-reaction" to "proactive prediction-reasoning," the framework allows AVs to anticipate human driver behaviors rather than merely reacting to them. The authors assert that their unified approach successfully bridges the gap between data-driven adaptability and physics-based safety guarantees, offering a robust solution for the complex, stochastic nature of mixed traffic flow decision-making.

More like this