Reinforcement Learning with Distributed MPC for Fuel-Efficient Platoon Control with Discrete Gear Transitions

Technical Summary: Reinforcement Learning with Distributed MPC for Fuel-Efficient Platoon Control with Discrete Gear Transitions

Problem Statement
Cooperative control of autonomous vehicle (AV) platoons offers significant potential for improving transportation efficiency and fuel economy through coordinated speed and powertrain management. While Model Predictive Control (MPC) is a prevalent approach for optimizing vehicle trajectories under constraints, the inclusion of discrete gear-shift decisions transforms the optimization problem from a continuous Nonlinear Program (NLP) into a Mixed-Integer Nonlinear Program (MINLP). Solving MINLPs in real-time is computationally intensive, often requiring excessive sample times or high-end hardware, which hinders online implementation for platoons. Existing alternatives, such as approximating the problem as a Mixed-Integer Quadratic Program (MIQP) or decoupling gear selection from speed control, often introduce significant suboptimality or fail to guarantee constraint satisfaction. Furthermore, previous learning-based approaches have been limited to single-vehicle scenarios or rely on supervised learning, which requires computationally expensive data generation from the very MINLP problems they aim to replace.

Methodology
The authors propose a Reinforcement Learning (RL)-based distributed MPC framework that decouples the discrete gear-shift schedule selection from the continuous speed and torque optimization. The core methodology involves two main components:

1. Parameterized Distributed MPC: For each vehicle in the platoon, a local MPC controller solves a continuous NLP over a prediction horizon. The discrete gear-shift schedule across this horizon is not optimized online but is instead fixed by a learned policy. This reduces the online optimization problem from an intractable MINLP to a tractable NLP. To ensure feasibility, the learned policy is combined with a heuristic gear-shift schedule (based on constant gear selection) in a parallel solving scheme; the controller applies the solution yielding the lowest cost, guaranteeing that a feasible solution is always available.

2. Reinforcement Learning Policy Architecture:

   • Single-Agent Training for Multi-Agent Generalization: To avoid the complexity and non-stationarity of Multi-Agent Reinforcement Learning (MARL), the policy is trained in a single-vehicle scenario tracking a reference trajectory. The policy input is parameterized using "shifted" variables (past predicted states, inputs, and desired trajectories) which implicitly encode information about the surrounding platoon dynamics. This allows a policy trained on a single vehicle to generalize to platoons of varying sizes.
   • Recurrent Neural Network (RNN): The policy utilizes a sequence-to-sequence RNN architecture. Instead of outputting a full gear sequence (which would cause the action space to grow exponentially with the prediction horizon), the RNN outputs a sequence of shift commands (down, no-shift, up-shift) iteratively. This structure ensures the input and output dimensions remain constant regardless of the prediction horizon length.
   • Two-Stage Training: The RL training employs a two-stage cost function. The first stage focuses on learning feasible gear-shift sequences to avoid infeasible NLP solutions. The second stage refines the policy to outperform the heuristic baseline, optimizing for fuel efficiency and tracking accuracy.

Key Contributions

• Computational Efficiency: By shifting the computational burden of optimizing gear-shift schedules offline via RL, the online controller only solves NLPs. This yields a significantly lower online computational burden compared to approaches solving mixed-integer problems.
• Performance via Co-optimization: Unlike decoupled approaches where gear selection is independent of speed control, the RL policy learns to select optimal gear-shift schedules across the entire MPC prediction window. This retains the benefits of co-optimization while avoiding the MINLP complexity.
• Scalable Policy Parameterization: The proposed architecture allows for training in a single-vehicle scenario while capturing inter-vehicle interactions through shifted state variables. This enables generalization to large platoons without retraining or increasing network size, and allows for parameter sharing in homogeneous vehicle fleets.
• Exponential Action Space Handling: The use of a recurrent architecture addresses the exponential growth of the action space with the prediction horizon, allowing the policy to generalize across different horizon lengths once trained.

Results
The approach was validated in highway-driving simulations involving platoons of up to five homogeneous vehicles. The proposed Learning Co-optimization (LC) controllers were compared against MINLP-based MPC (baseline), MIQP-based MPC, and various heuristic decoupled or co-optimization strategies.

• Performance: The LC controller trained with both stages (LC-2) achieved performance very close to the computationally intensive MINLP baseline, with a relative cost increase of approximately 1.7% to 2.8% in single and five-vehicle scenarios, respectively. This significantly outperformed decoupled heuristics (HD) and showed superior consistency (lower standard deviation) compared to MIQP and other heuristics.
• Computation Time: The proposed approach demonstrated a computational speedup of one to two orders of magnitude over mixed-integer approaches. For a five-vehicle platoon, the LC-2 controller solved the problem in approximately 0.63 seconds per time step, whereas the MINLP approach required over 2200 seconds (and was often infeasible within reasonable time limits for longer horizons).
• Generalization: The policy trained on a single vehicle successfully generalized to a five-vehicle platoon. Furthermore, the approach maintained performance when the prediction horizon was increased from 15 to 30 time steps, whereas mixed-integer approaches became computationally intractable.

Significance
The paper claims that this work presents a novel learning-based MPC scheme that effectively addresses the computational intractability of fuel-efficient platoon control with discrete gear transitions. By isolating the discrete component via RL and retaining the continuous co-optimization via distributed NLP, the approach achieves a balance between optimality and real-time feasibility. The significance lies in demonstrating that RL policies trained in simplified single-agent environments can generalize to complex multi-agent cooperative control tasks, offering a scalable solution that significantly reduces online computational burden while maintaining high fuel efficiency and tracking performance comparable to the theoretical MINLP optimum.