Transfer Learning for Customized Car Racing Environments

Technical Summary: Transfer Learning for Customized Car Racing Environments

Problem Statement
Reinforcement Learning (RL) faces a fundamental trade-off between exploration and exploitation, often requiring vast amounts of data to converge, particularly in environments with sparse rewards, partial observability, and high-dimensional continuous control spaces. While Transfer Learning (TL) is well-established in supervised deep learning, its application in deep RL is non-trivial due to the complexity of transferring knowledge within Markov Decision Processes (MDPs). The specific challenge addressed in this work is achieving fast lap times in OpenAI's Car Racing environment by training an agent on a source circuit and applying that knowledge to customized target environments (featuring different maps, acceleration, braking, or friction parameters) via zero-shot transfer or fine-tuning.

Methodology
The study employs a comparative analysis of model-free and model-based deep reinforcement learning algorithms within the OpenAI Gym Car Racing environment.

• Algorithms Tested:
  • Model-Free: Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), and Deep Deterministic Policy Gradient (DDPG).
  • Model-Based: Dreamer (specifically a customized DreamerV2), which learns latent dynamics through a world model to plan in "imagination."
• Experimental Setup:
  • Source Domain: A standard Car Racing map with less extreme turns, used for initial training over 1 million steps.
  • Target Domains: Customized environments created by altering:
    1. Race Map: A target map with tighter, more difficult turns.
    2. Dynamics: Modified acceleration (+0.2/timestep vs. default +0.1), braking coefficients (0.6 vs. 0.8), and grass friction (0.8 vs. 0.6).
  • Transfer Protocols:
    • Zero-Shot Transfer: Evaluating pre-trained models on target domains without further interaction.
    • Few-Shot Transfer (Fine-tuning): Additional training of 300k steps on the target domain.
  • Implementation: The authors utilized the Stable Baselines3 (SB3) repository for PPO, SAC, and DDPG, and a customized fork of the DreamerV2 repository. Neural network weights were transferred between source and target tasks.

Key Contributions
The paper presents an extensive ablation study comparing the efficacy of model-based versus model-free approaches in transfer learning scenarios for continuous control tasks. The specific contributions include:

1. Systematic testing of PPO, SAC, DDPG, and Dreamer on the standard OpenAI CarRacing environment.
2. Evaluation of these algorithms' ability to transfer to customized environments involving changes in race tracks and vehicular dynamics (acceleration, braking, friction).
3. A comparative analysis of convergence speed, sample efficiency, and final performance between model-based and model-free paradigms in the context of transfer learning.

Results

• Base Environment Performance: The model-based approach (Dreamer) significantly outperformed model-free counterparts, achieving rewards over 900 and converging within approximately 200k steps. In contrast, PPO and SAC reached rewards of 550 and ~500 respectively at 1M steps. DDPG failed to converge, consistently yielding negative rewards (-84) despite extensive hyperparameter tuning.
• Zero-Shot Transfer: On the target map, Dreamer achieved the highest reward (500), followed by PPO (400) and SAC (~350). All performed below their source map levels. DDPG continued to struggle.
• Fine-Tuning: After 300k steps of fine-tuning, Dreamer recovered to source-level performance (~915 rewards) and solved the target task. PPO and SAC showed performance similar to their zero-shot counterparts, failing to match Dreamer's recovery.
• Customized Dynamics:
  • Acceleration: PPO and Dreamer maintained base-level performance; SAC dropped slightly.
  • Braking: PPO and SAC performed worse than the base; Dreamer maintained base-level performance.
  • Friction: PPO performed significantly worse and degraded further with fine-tuning; SAC maintained base levels; Dreamer performed slightly worse but still outperformed all model-free algorithms.
• Qualitative Observations: Video analysis revealed that while model-free agents sometimes achieved decent rewards, they often exhibited unsafe behaviors (e.g., driving in the wrong direction or skidding). Dreamer, particularly after fine-tuning, demonstrated superior maneuvering through sharp turns.

Significance and Claims
The authors claim that model-based approaches, specifically Dreamer, dominate in this environment regarding performance, convergence speed, and sample efficiency compared to model-free methods. The study highlights that model-free algorithms exhibit brittle convergence properties, requiring meticulous hyperparameter tuning and still failing to generalize reliably across different environments (as seen with DDPG and the degradation of PPO in high-friction scenarios).

The paper asserts that transfer learning, particularly when combined with fine-tuning, not only boosts performance on target domains but also demonstrates high performance ability during the learning process, suggesting potential for safe reinforcement learning. The authors conclude that while they successfully solved the car racing task in customized setups using model-based RL, their work was constrained by computational limits (each experiment taking ~24 hours). They express interest in future work involving meta-RL, safe-RL, and real-world testing, but do not claim immediate applicability beyond the scope of their simulation experiments.