⚡ electrical engineering

Platooning Connected, Autonomous, and Human-Driven Vehicles: A Deep Reinforcement Learning-based Approach

This paper proposes a deep reinforcement learning-based hybrid platooning control strategy that dynamically integrates non-connected vehicles into mixed traffic to enhance flexibility while suppressing disturbance propagation and optimizing the trade-off between traffic throughput and stability.

Original authors: Zhen Qina, Dong-Fan Xie, Heng Ma, Xiaomei Zhao, Zhengbing He

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhen Qina, Dong-Fan Xie, Heng Ma, Xiaomei Zhao, Zhengbing He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a highway as a long river of cars. For a long time, scientists have tried to make this river flow smoother by teaching the "smart" cars (those with computers and internet connections) to drive in tight, synchronized groups called platoons. Think of a platoon like a train of cars moving together with very little space between them, which saves fuel and moves more cars through the same road.

However, there's a big problem: right now, most cars on the road aren't "smart" yet. They are regular human-driven cars or older self-driving cars that can't talk to each other. Traditional platooning strategies ignore these "dumb" cars, which doesn't reflect reality.

This paper proposes a new, more flexible way to handle traffic: letting the "dumb" cars join the "smart" train, but with a smart manager.

Here is how the researchers solved the puzzle, explained simply:

1. The Problem: The "Too-Long Train" Trap

The researchers realized that if you just let any car join a platoon whenever it wants, the trains get too long.

The Analogy: Imagine a line of people holding hands. If you let everyone join the line instantly without checking, the line becomes a giant, wobbly snake. If the person at the very front stumbles, the person at the very back gets jerked violently. In traffic, this means a small brake tap by the lead car causes a massive, dangerous "shockwave" of braking all the way down the line, leading to traffic jams and accidents.

2. The Solution: The "Deep Reinforcement Learning" Coach

To fix this, the authors created a digital "Coach" using a type of Artificial Intelligence called Deep Reinforcement Learning (DRL).

How it works: Imagine a coach watching a game of soccer. The coach doesn't just tell players to "run fast." Instead, the coach watches the whole field, sees where the players are, and decides: "Okay, let's form a small group here, but break up that group over there to keep things stable."
The Magic: This AI coach learns by trial and error. It tries different ways to group the cars (some groups of 3, some of 5, some mixing smart and dumb cars). It gets a "reward" when traffic flows smoothly and fuel is saved, and a "penalty" when traffic gets bumpy or dangerous. Over millions of practice runs (simulations), it learns the perfect balance.

3. The Rules of the New Game

The paper sets up a specific set of rules for this new "Hybrid Platoon":

The "Smart" Bridge: A regular car (Human-driven or non-connected) can join the platoon, but only if it is sandwiched between two "smart" cars (Connected Autonomous Vehicles). The smart cars act like translators, sensing the regular car's speed and passing that info to the rest of the group.
No Solo Leaders: A regular car cannot be the very first or very last car in the platoon because it can't talk to the group to say, "I'm leading!" or "I'm the tail!"
The Limit: You can't have a long chain of regular cars in the middle. If there are too many regular cars in a row without a smart car to "bridge" the gap, the platoon breaks up. This prevents the "giant wobbly snake" problem.

4. What Happened in the Simulations?

The researchers ran thousands of computer simulations to see if their "Coach" worked. Here is what they found:

Stability: The AI-controlled platoons were much steadier. When the lead car braked, the shockwave didn't get amplified as much. The traffic flowed like a calm river rather than a choppy sea.
Safety: The risk of collisions (measured by how close cars got to hitting each other) dropped significantly, especially when there weren't many smart cars on the road yet.
Efficiency: While the AI sometimes chose to make slightly smaller groups to keep things safe, it still managed to move a lot of cars. It found a "sweet spot" where the road was safe and efficient.
Cleaner Air: Because the cars weren't braking and accelerating as wildly, they burned less fuel and emitted less pollution (like CO2 and NOx).

The Bottom Line

This paper doesn't just say "let's put smart cars in groups." It says, "Let's mix smart and regular cars, but use a super-smart AI coach to decide exactly how big those groups should be and who can join, so we don't create traffic chaos."

The result is a traffic system that is safer, more stable, and cleaner, even before every single car on the road becomes fully autonomous. The AI acts as the conductor of a traffic orchestra, ensuring that even if some instruments (cars) are a bit out of tune, the whole symphony still plays smoothly.

Technical Summary: Platooning Connected, Autonomous, and Human-Driven Vehicles via Deep Reinforcement Learning

1. Problem Statement

Existing vehicle platooning strategies are predominantly designed for homogeneous environments consisting solely of Connected Autonomous Vehicles (CAVs) or Connected Human-driven Vehicles (CHVs). These approaches typically exclude non-connected vehicles, such as Autonomous Vehicles (AVs) and Human-driven Vehicles (HVs), thereby failing to accurately reflect current and near-future mixed traffic conditions.

While recent "hybrid" strategies attempt to incorporate non-connected vehicles into platoons by leveraging CAVs as information relays, they often suffer from a critical limitation: the unregulated integration of HVs/AVs can trigger rapid platoon expansion. This uncontrolled growth amplifies the risk of disturbance propagation within the traffic flow, exacerbating the inherent conflict between maximizing traffic throughput and maintaining flow stability. Furthermore, conventional constraints on maximum platoon length, designed for homogeneous fleets, are insufficient for these emerging heterogeneous scenarios. The core challenge is to develop a control strategy that allows for the flexible inclusion of non-connected vehicles while dynamically balancing traffic capacity against flow stability and safety.

2. Methodology

The study proposes a Deep Reinforcement Learning (DRL)-based hybrid platooning control strategy designed to operate within a Partially Observable Markov Decision Process (POMDP) framework.

2.1. Hybrid Platooning Pattern

The authors introduce a "hybrid platooning pattern" (Scheme PS2) that conditionally permits non-connected vehicles (HVs/AVs) to join platoons.

Mechanism: CAVs utilize onboard multi-sensor perception (LiDAR, radar, cameras) to sense the states of adjacent non-connected vehicles. This information is then broadcast via Vehicle-to-Everything (V2X) communication to other connected members, rendering the states of non-connected vehicles observable within the platoon.
Constraints: To ensure platoon integrity, HVs/AVs are prohibited from acting as platoon leaders or tail vehicles. A platoon can only form if the state of every vehicle is observable to connected members. This limits the maximum number of consecutive non-connected vehicles within a platoon to two (e.g., CAV–HV–HV–CAV).
Comparison: This is contrasted with Scheme PS1, where only CHVs/CAVs form platoons, and non-connected vehicles are excluded.

2.2. DRL Framework

The control strategy employs a Centralized Training with Decentralized Execution (CTDE) architecture using the Proximal Policy Optimization (PPO) algorithm.

State Space: The state representation integrates multi-modal data, including ego-vehicle dynamics (type, platoon position, time headway) and ambient environment states (types and positions of the two immediate preceding and following vehicles). This design captures local topology without succumbing to the curse of dimensionality.
Action Space: Discrete actions include joining a platoon, leaving a platoon, or maintaining independent driving.
Reward Function: A multi-objective reward function is formulated to balance three components:
1. Traffic Efficiency ( $r_{capacity}$ ): Based on the percentage increase in lane capacity relative to a baseline.
2. Traffic Stability ( $r_{stability}$ ): Based on the improvement of a hybrid traffic flow stability discriminant ( $G_h$ ), derived from car-following model derivatives.
3. Penalty ( $r_{penalty}$ ): A constraint penalty applied if lane capacity drops significantly below the baseline, preventing the agent from sacrificing efficiency for excessive stability.
Training: The policy network learns to dynamically adjust platoon structures (size and distribution) to optimize the trade-off between capacity and stability.

3. Key Contributions

Hybrid Platooning Pattern: The paper defines a feasible mechanism for integrating non-connected vehicles into platoons via CAV-mediated information propagation, addressing the scalability limitations of strictly connected fleets.
DRL-Based Control Strategy: It develops a novel DRL strategy that moves beyond fixed formation rules. Instead of assuming a static platoon composition, the strategy learns to dynamically optimize platoon structures in real-time to manage the trade-off between throughput and stability.
Multi-Level State Representation: The study integrates vehicle dynamics, platoon topology, and traffic flow states into a unified state representation, enabling agents to make decentralized decisions based on local observations while considering global stability goals.
Comprehensive Evaluation: The approach is rigorously evaluated against traditional connected platooning (PS1) and unregulated hybrid platooning (PS2) across four dimensions: generalization capability, traffic flow stability, safety, and environmental impact.

4. Experimental Results

Numerical simulations were conducted on a single-lane road with 1,000 vehicles under varying CAV penetration rates ( $P_{CAV}$ ).

Training Performance: The DRL strategy successfully converged, achieving a 27.9% improvement in the traffic stability discriminant ( $G_h$ ) compared to the baseline hybrid strategy (PS2). It achieved this by reducing the maximum platoon length by 28.3% (to ~24 vehicles) and increasing the total number of platoons by 19.3%, effectively distributing traffic density.
Generalization: The strategy demonstrated robust generalization across $P_{CAV}$ values from 0.1 to 0.5. When $P_{CAV} > 0.1$ , the DRL strategy improved the stability discriminant by 51.08% compared to both PS1 and PS2. Unlike fixed strategies, which suffer from exponential growth in platoon size at higher penetration rates, the DRL strategy maintained average platoon lengths at approximately 35 vehicles.
Disturbance Propagation: In scenarios involving a lead-vehicle deceleration disturbance, the DRL strategy significantly suppressed velocity oscillations. It reduced the peak average Time-to-Collision Inverse (TTCI) by 52.95% compared to conventional platooning (PS1) under low CAV penetration ( $P_{CAV}=0.1$ ).
Safety and Efficiency: The strategy reduced peak velocity standard deviation by up to 38.60% compared to PS1. In terms of environmental impact, under low CAV penetration, the DRL strategy reduced peak average fuel consumption by 1.90%, CO2 emissions by over 1.2%, and NOx emissions by up to 7.1% compared to conventional strategies.

5. Significance and Claims

The authors claim that the proposed DRL-based approach effectively resolves the conflict between traffic throughput and stability in mixed traffic environments. By dynamically optimizing platoon structures rather than adhering to rigid formation rules, the strategy suppresses velocity disturbance propagation and enhances the safety of mixed traffic flows.

The study emphasizes that while traditional hybrid strategies (PS2) maximize capacity by allowing large, unregulated platoons, they compromise stability. Conversely, the DRL strategy achieves a "nonlinear equilibrium," accepting a marginal reduction in peak capacity (approx. 4.6% compared to PS2 in training) to achieve substantial gains in stability and safety. The paper concludes that this approach offers a robust solution for the transitional period of mixed traffic, where non-connected vehicles remain prevalent, and suggests that future work should address multi-lane interactions, communication failures, and dynamic platoon splitting.