⚡ electrical engineering

Integrated Automated Car Following and Lane-changing control based on a Parametrized Deep Q-network with Hybrid Action Space

This paper proposes an integrated control framework for connected and automated vehicles that utilizes a Parametrized Deep Q-Network with a hybrid action space to simultaneously optimize discrete lane-changing decisions and continuous acceleration, thereby outperforming traditional separated control methods in safety and comfort.

Original authors: Hao Zhang, Zihao Li, Yang Zhou

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Zhang, Zihao Li, Yang Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine driving a car that doesn't just follow the rules of the road but actually thinks about how to drive better, safer, and smoother. This paper presents a new "brain" for self-driving cars that solves a specific problem: how to decide when to change lanes and how to speed up or slow down at the exact same time.

Here is the breakdown of the paper's ideas using simple analogies:

The Problem: The "Two-Headed" Driver

Traditionally, self-driving cars have been taught to handle two tasks separately, like a driver with two different brains:

Brain A (Lane Changing): Decides when to switch lanes. It looks at the traffic and says, "Okay, I'll move left."
Brain B (Car Following): Decides how fast to go. It says, "I need to speed up to catch that gap" or "I need to brake to stay safe."

The Flaw: These two brains often don't talk to each other.

The Paper's Example: Imagine you want to move to the left lane. There is a gap between two cars, but it's a bit tight.
- The Old Way: Brain A sees the gap is too small right now and says, "No, don't move." Brain B just keeps driving at the current speed. You miss the opportunity to change lanes because you didn't speed up to fit into the gap.
- The Real World: A human driver would think, "I'll hit the gas for a second to squeeze into that gap, then move over."

The paper argues that these two decisions (changing lanes and adjusting speed) are intertwined. You can't make a good lane change decision without knowing how you will adjust your speed to make it happen.

The Solution: The "Parametrized Deep Q-Network" (P-DQN)

The authors created a new AI system called P-DQN. Think of this as a Master Chef who manages a kitchen with two types of ingredients:

Discrete Ingredients (The "Yes/No" decisions): Like deciding which lane to move to (Left, Stay, or Right). This is a clear, single choice.
Continuous Ingredients (The "How much" decisions): Like deciding exactly how much to press the gas or brake pedal. This is a smooth, sliding scale, not just "on" or "off."

Most old AI systems struggled to mix these two. They tried to turn the smooth "gas pedal" into a list of fixed buttons (e.g., "Press 10%," "Press 20%"), which is clumsy and inefficient.

How P-DQN Works:

The High-Level Manager (The Discrete Part): This part of the AI looks at the traffic and decides, "We are going to move to the left lane."
The Low-Level Worker (The Continuous Part): Once the decision is made, this part instantly figures out the exact amount of acceleration needed to make that move safely and smoothly.
The Magic: The Manager and the Worker talk to each other instantly. If the Manager says "Move Left," the Worker immediately calculates, "Okay, I need to speed up 2 meters per second to fit in that gap."

How It Learned (The "Reward System")

The AI learned through trial and error, similar to training a dog with treats. The researchers gave the AI a "scorecard" (Reward Function) with four main goals:

Safety (The Big Penalty): If you hit a car or the wall, you get a huge negative score.
Following the Leader: If you stay at a comfortable distance behind the car in front of you, you get points.
Staying in the Lane: If you drive down the middle of your lane without wobbling, you get points.
Getting to the Right Lane: If you successfully move to a faster lane to go faster, you get points.

The AI played thousands of virtual driving simulations, trying to get the highest total score. Over time, it learned that sometimes the best way to get the "Lane Change" points was to first speed up (the "Car Following" action) to create a safe gap.

The Results: The New AI vs. The Old Rules

The researchers tested their new AI against a standard, rule-based system (called MOBIL + IDM), which is like a car following a strict instruction manual.

Comfort: The new AI drove much smoother. It didn't jerk the car around as much because it planned the speed changes before the lane change.
Safety: The new AI was much safer. It kept a better distance from other cars. The old system sometimes got too close to other vehicles because it didn't coordinate its speed with its lane change.
Efficiency: The new AI could successfully change lanes in situations where the old system would just give up and stay stuck in the slow lane.

In Summary

This paper introduces a smarter way to teach self-driving cars. Instead of treating "changing lanes" and "adjusting speed" as two separate chores, the new system treats them as one coordinated dance. By using a special type of AI (P-DQN) that can handle both "Yes/No" decisions and "How much" adjustments simultaneously, the car learns to drive more like a skilled human: accelerating to find a gap, then smoothly sliding into it, all while keeping everyone safe and comfortable.

Technical Summary: Integrated Automated Car Following and Lane-changing Control based on a Parametrized Deep Q-network

Problem Statement
Current approaches to automated vehicle control often treat lane-changing and car-following as independent tasks. Traditional rule-based models, such as the Minimizing Overall Braking Decelerations Induced by Lane Changes (MOBIL) combined with the Intelligent Driver Model (IDM), rely on pre-defined rules and explicit trajectory approximations. These methods struggle to handle the dynamic and stochastic nature of traffic, particularly when lane-changing decisions depend on complex interactions with surrounding vehicles (e.g., adjusting speed to utilize an acceptable gap). Furthermore, existing Reinforcement Learning (RL) approaches often address these subtasks separately or attempt to integrate them by discretizing continuous actions (like acceleration), which can lead to a loss of the natural structure of the action space and reduced efficiency. The core challenge lies in managing a hybrid action space where lane-change decisions are discrete, while acceleration and trajectory execution are continuous.

Methodology
The study proposes an integrated control framework for Connected and Automated Vehicles (CAVs) utilizing a Parametrized Deep Q-Network (P-DQN). This algorithm is designed to operate directly on a hybrid discrete-continuous action space without discretizing continuous variables.

State Space: The system state ( $S_t$ $S_{t}$ ) comprises two components:
1. Vehicle Interactions ( $S^1_t$ ): Includes relative lateral/longitudinal positions and velocities of four surrounding vehicles (leading/following in the current and adjacent lanes).
2. Road Context ( $S^2_t$ ): Includes the ego vehicle's distance to road boundaries, the ideal lane centerline, and the lane index difference between the current and ideal lanes.
Action Space: The framework employs a hybrid action space $A_t = [A^h_t, A^l_t]^T$ $A_{t} = [A_{t}^{h}, A_{t}^{l}]^{T}$ :
- High-level Discrete Action ( $k_t$ ): Determines the lane-change decision (0: left, 1: stay, 2: right).
- Low-level Continuous Actions ( $A^l_t$ ): Controls lateral ( $a_{x,0}$ ) and longitudinal ( $a_{y,0}$ ) accelerations. The range of continuous actions is parameterized by the discrete decision (e.g., lateral acceleration is constrained to negative values for a left lane change).
Reward Function: The total reward ( $R_{total}$ $R_{t o t a l}$ ) is a weighted sum of five components designed to balance safety, efficiency, and comfort:
1. Safety: Penalizes proximity to surrounding vehicles and road boundaries using logarithmic penalty functions.
2. Car-Following: Encourages maintaining a desired time gap with the leading vehicle.
3. Lane-Keeping: Penalizes deviation from the lane centerline when not changing lanes.
4. Lane-Changing: Rewards alignment with the "ideal lane" (determined by the speed of leading vehicles in adjacent lanes).
Algorithm: The P-DQN utilizes a critic network to estimate the action-value function $Q(S, k, A_l)$ and an actor network to output the optimal continuous parameters $A_l$ for a given discrete action $k$ . The network is trained to maximize cumulative rewards using a least-squares loss function and stochastic gradient descent.

Key Contributions

Integrated Control Framework: The study presents a unified control strategy that simultaneously manages lane-changing and car-following, treating them as interdependent processes rather than isolated subtasks.
Hybrid Action Space Implementation: The paper implements P-DQN to handle the specific challenge of hybrid action spaces in CAV control, avoiding the limitations of discretizing continuous actions.
Multi-Objective Optimization: The framework successfully integrates four sub-objectives—collision avoidance, car-following, lane-keeping, and lane-changing efficiency—into a single learning policy.

Results
Numerical experiments were conducted across four scenarios with varying complexities, including constant and dynamic speeds of surrounding vehicles.

Convergence: The P-DQN algorithm achieved convergence within 2,000 to 3,000 training episodes.
Behavioral Performance: Unlike traditional methods (e.g., MOBIL) that might reject a lane change due to immediate gap constraints, the P-DQN learned to adjust longitudinal speed (accelerate or decelerate) to create or utilize gaps, enabling successful lane changes. The agent maintained safe distances from both leading and trailing vehicles during maneuvers and adhered to lane centerlines.
Comparative Analysis: When compared against a baseline MOBIL+IDM model:
- Safety: P-DQN demonstrated significantly lower Inverse Time-to-Collision (Inverse-TTC) values (e.g., 0.075 vs. 0.214 in Scenario 1), indicating a lower risk of collision.
- Comfort: P-DQN achieved substantially lower driving discomfort metrics (based on acceleration magnitude), with values ranging from 0.599 to 1.260 compared to 1.596 to 3.712 for the MOBIL+IDM baseline.

Significance and Claims
The paper claims that the proposed P-DQN framework resolves the limitations of separate control strategies and rule-based models by effectively integrating lane-changing and car-following behaviors. By leveraging a hybrid action space, the model enables CAVs to make coordinated decisions where speed adjustments facilitate safe lane changes, mimicking realistic human driving behaviors. The study concludes that this integrated approach yields superior performance in terms of both driving safety and comfort compared to the established combination of MOBIL and IDM, providing a more adaptable solution for complex traffic scenarios without relying on rigid pre-defined rules.