Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control

1. Problem Statement

The paper addresses a fundamental challenge in robotics and autonomous control: learning high-performance policies that remain consistent with expert human behavior.

• The Tension: Pure Reinforcement Learning (RL) often discovers strategies that maximize task performance (e.g., fastest lap time) but deviate significantly from human driving styles, potentially exploiting simulator quirks or adopting unrealistic control patterns. Conversely, Imitation Learning (IL) is limited by the quality of demonstrations and struggles to improve beyond the expert's performance or adapt to new conditions (covariate shift).
• Specific Context: This is critical in high-performance motorsports, where vehicles operate near physical limits. Small deviations in control strategy affect both stability and performance. Furthermore, the consequences of decisions (e.g., entry speed) are often delayed, making temporal credit assignment difficult.
• Goal: Develop a framework that optimizes task performance (lap time) while strictly enforcing a constraint on behavioral consistency with professional drivers, enabling the policy to act as a reliable "digital surrogate" for human evaluation in virtual setup testing.

2. Methodology

The authors propose a Behavior-Constrained Reinforcement Learning (BCRL) framework tailored for racing, integrating three core components:

A. Constrained Optimization Formulation

Instead of simply weighting performance and imitation rewards (which requires manual tuning of trade-off coefficients), the authors formulate the problem as a constrained optimization task:

• Objective: Maximize expected cumulative performance reward ($r_p$).
• Constraint: Enforce a lower bound ($\hat{R}$) on the expected style imitation reward ($r_s$) at all times.
• Mechanism: The style imitation coefficient ($\alpha_s$) is treated as a Lagrangian multiplier. It is dynamically updated during training based on the gap between the current imitation performance and the target bound $\hat{R}$. This eliminates the need for extensive hyperparameter tuning across different environments.

B. Receding-Horizon Credit Assignment

To address the delayed consequences of control decisions in dynamic systems:

• Trajectory Predictor: A lightweight auxiliary network predicts short-term future trajectories using probabilistic Bézier curves. This reduces dimensionality while preserving geometric structure.
• Look-Ahead Rewards: The predictor generates a distribution of future states. The policy receives "look-ahead" rewards based on the expected progress and style adherence of these predicted trajectories. This improves temporal credit assignment by providing immediate feedback for actions that will only manifest as success/failure seconds later.

C. Contextual Policy and Data Augmentation

• Contextual MDP: The policy is conditioned on a reference trajectory ($\tau$). This allows the policy to learn a distribution of valid behaviors rather than a single deterministic target, accounting for natural human variability under disturbances.
• Reference Trajectory Distribution (RTD): To improve robustness and generalization, the authors fit a distribution (using Probabilistic Movement Primitives, ProMP) to expert demonstrations. During training, the policy is conditioned on samples drawn from this distribution rather than fixed trajectories, effectively augmenting the data and preventing overfitting to specific lines.
• Training Pipeline:
  1. Supervised Pretraining: The policy and predictor are initialized via Behavioral Cloning (BC) on expert data.
  2. Fine-tuning: Proximal Policy Optimization (PPO) is used with the constrained objective. The predictor is updated before the policy to ensure rewards are based on the most recent model.

3. Key Contributions

1. Behavior-Constrained Policy Optimization: A novel formulation that treats demonstrations as a hard constraint (via a lower bound on imitation error) rather than a soft reward, ensuring strict adherence to expert behavior while optimizing performance.
2. Receding-Horizon Credit Assignment: Introduction of a probabilistic Bézier curve predictor that provides look-ahead rewards, significantly improving learning efficiency and temporal reasoning in high-speed dynamic control.
3. Trajectory-Conditioned Modeling: A method to represent the natural variability of expert driving by conditioning the policy on a distribution of reference trajectories, enhancing robustness to setup changes.
4. Human-Grounded Validation: A comprehensive evaluation pipeline using a Driver-in-the-Loop (DiL) simulator, validating that the learned policies not only match lap times but also reproduce setup-dependent driving characteristics consistent with feedback from professional race drivers.

4. Experimental Results

The method was evaluated in a high-fidelity race car simulation using data from professional drivers.

• Performance vs. Imitation Trade-off:
  • The proposed method (Setting A) achieved the best Pareto frontier, outperforming baselines (Standard Step Rewards, Adversarial Motion Priors, and standard Contextual RL) in both lap time and Mean Reference Offset (MRO).
  • It successfully balanced performance and style without manual tuning of the trade-off coefficient, whereas baselines required extensive grid searches and still failed to match the proposed method's balance.
• Generalization:
  • The method maintained top performance and imitation quality across three different race tracks and various vehicle configurations without re-tuning, demonstrating superior robustness compared to baselines.
• Ablation Studies:
  • Automatic Scaling: The dynamic adjustment of $\alpha_s$ proved superior to fixed coefficients, achieving target imitation quality with minimal performance loss.
  • Data Augmentation: Using RTD-based augmentation significantly improved generalization on test tracks compared to training on raw demonstrations directly.
  • Prediction Module: Including the trajectory predictor as an auxiliary task improved lap times by ~0.25s and MRO by ~30%, confirming the value of look-ahead credit assignment.
• Human-Grounded Evaluation:
  • In a DiL study, the learned policies correctly reproduced the qualitative handling characteristics (e.g., understeer vs. oversteer responses) reported by professional drivers for different vehicle setups.
  • The policies identified the most "drivable" setup (best balance of speed and consistency), aligning with expert engineering intuition.

5. Significance

This work bridges the gap between pure performance optimization and behavioral fidelity in high-stakes control systems.

• Practical Application: It provides a reliable tool for virtual setup testing in motorsports. Engineers can use these "digital drivers" to evaluate vehicle configurations (suspension, aerodynamics) without needing a physical car or a human driver for every iteration, saving time and cost.
• Broader Impact: The framework extends beyond racing to any domain requiring continuous control that must adhere to human-like constraints (e.g., autonomous driving, robotic manipulation), offering a structured approach to safe, interpretable, and high-performance RL.
• Methodological Advance: The combination of constrained optimization, receding-horizon prediction, and distributional conditioning offers a robust template for solving multi-objective control problems where "how" the task is done is as important as "how well" it is done.