Enhanced Deep Q-Learning for 2D Self-Driving Cars: Implementation and Evaluation on a Custom Track Environment

This paper presents the implementation and evaluation of an enhanced Deep Q-Learning algorithm with a priority-based action selection mechanism for a 2D self-driving car on a custom Pygame track, demonstrating a 60% improvement in average reward over the original DQN after 1000 training episodes.

The Gist

Imagine you are teaching a toddler how to ride a bicycle on a winding path. You don't give them a manual with physics equations; instead, you let them ride, and every time they wobble or hit a tree, you give them a gentle "ouch" (a penalty). Every time they stay upright and move forward, you give them a high-five (a reward). Eventually, they learn the best way to pedal and steer without falling.

This paper is about doing exactly that, but with a computer program acting as the "toddler" and a video game acting as the "bicycle path."

Here is the breakdown of their project in simple terms:

1. The Playground: A Digital Map

The researchers didn't want to crash real cars (too expensive and dangerous!). Instead, they built a video game using a tool called Pygame.

• The Track: They drew a map that looks like the roads around the University of Memphis.
• The Car: A simple digital sprite (an image) that moves forward automatically. It can't speed up or brake; it just moves.
• The Eyes (Sensors): Imagine the car has 7 laser beams shooting out from its front, like a spider's web. These beams measure how far away the walls are. If a beam hits a wall, it's short. If the road is clear, the beam is long. This is the only information the car "sees."

2. The Teacher: Reinforcement Learning

The car learns through a method called Reinforcement Learning. Think of it as a game of "Hot and Cold."

• The Goal: Drive around the whole track without crashing.
• The Rules:
  • If the car stays on the road: +5 points (High five!).
  • If the car hits a wall: -20 points (Ouch!).
• The Choices: The car can only do three things: Turn Left, Turn Right, or Go Straight.

3. The Brain: Three Different Students

The researchers tested three different "brains" (algorithms) to see which one could learn to drive the track best.

• Student A: The Vanilla Neural Network

  • Analogy: A smart kid who learns by trial and error but doesn't have a specific strategy.
  • Result: It eventually learned to drive the track, but it took a long time to figure things out. It was like a student who gets the right answer but takes forever to study.

• Student B: The Original DQN (Deep Q-Learning)

  • Analogy: A student with a powerful memory bank who tries to predict the future. It remembers every time it crashed and tries to avoid that situation next time.
  • Result: Surprisingly, this "smart" student struggled. It got stuck in loops and couldn't finish the track. It was overthinking the problem and getting confused.

• Student C: The Modified DQN (The Winner)

  • Analogy: This is the original smart student, but with a coach whispering in its ear.
  • The Secret Sauce: The researchers added a "Priority Rule." If the left sensor sees a wall coming close, the coach says, "Hey, turn right immediately!" If the right sensor sees a wall, "Turn left!"
  • Result: This combination was a home run. The car learned 60% faster and got a much higher score than the others. It finished the track smoothly.

4. The Hardware: The Gym

Training these digital brains is heavy lifting.

• They tried training on a standard laptop (CPU), which was like trying to run a marathon while carrying a heavy backpack. It took 12 hours.
• They switched to a powerful computer with a dedicated graphics card (GPU), which is like having a personal trainer and a treadmill. It finished the same training in just 4 hours.

The Big Takeaway

The paper proves that while powerful AI algorithms (like DQN) are great, they sometimes need a little help from simple, common-sense rules.

The Metaphor:
Imagine you are teaching a robot to walk through a minefield.

• The Old Way: Let the robot stumble around until it figures out the pattern. (Slow and risky).
• The New Way: Give the robot a metal detector (the sensors) and a rule: "If the detector beeps on the left, step right."
• The Result: The robot doesn't just learn by accident; it learns by combining its "brain" (AI) with a simple "reflex" (the priority rule).

Conclusion:
The researchers successfully built a self-driving car simulator where an AI learned to drive a custom track. By adding a simple "priority" rule to the AI's decision-making, they made it drive much better and faster than the standard AI models. It's a step toward making real self-driving cars that are safer and smarter.

Technical Summary

1. Problem Statement

The research addresses the challenge of training autonomous agents to navigate dynamic environments without human intervention. Specifically, the authors aim to overcome the limitations of standard Reinforcement Learning (RL) and traditional Q-learning in self-driving scenarios:

• State Space Complexity: Standard Q-learning fails in environments with large or continuous state spaces because it requires excessive storage for Q-values and cannot infer values for unseen states.
• Training Efficiency and Safety: Training in real-world scenarios is dangerous and time-consuming. While simulators exist, standard Deep Q-Networks (DQN) often struggle to converge efficiently or learn robust policies for complex tracks (e.g., sharp turns) within a reasonable timeframe.
• Action Selection: The authors identified that standard DQN agents sometimes struggle with precise steering in narrow tracks, leading to collisions or getting stuck.

2. Methodology

A. Simulation Environment

• Platform: The environment was built using Pygame (Python), creating a 2D simulation.
• Map: A custom track was designed to mimic the University of Memphis campus, derived from OpenStreetMap data and converted into a sprite-based image where transparent areas represent the drivable track and opaque areas represent obstacles.
• Vehicle Dynamics:
  • The car moves at a constant speed (throttle and braking were removed to simplify the action space).
  • Controls: The agent has a discrete action space of three options: Steer Left, Steer Right, and Go Straight (Do Nothing).
• Sensors: The car is equipped with 7 sensors positioned at the front, spaced 20 degrees apart.
  • Input State: The sensors measure the distance to the nearest obstacle in their respective direction.
  • Normalization: Distance values are normalized (divided by a max length of 1000) to create a state vector of 7 floating-point numbers.

B. Reward Function

The reward system is designed to encourage continuous movement and penalize collisions:

• No Collision: +5 reward per step.
• Collision: -20 reward (immediate termination of the episode).
• Goal: Maximize cumulative reward over an episode.

C. Algorithms Implemented

The study compares three approaches:

1. Vanilla Neural Network: A baseline feedforward network without the DQN architecture (no replay buffer or target network).
2. Standard DQN: A Deep Q-Network using TensorFlow with:
   • Architecture: 3 Dense layers (Input: 7, Hidden: 64, 64, Output: 3).
   • Components: Experience Replay Buffer (size 3000), Target Network (updated periodically), and Epsilon-Greedy exploration strategy.
   • Hyperparameters: Discount factor ($\gamma$) = 0.99, Learning rate = 0.001, Batch size = 128.
3. Modified DQN (Priority-Based): An enhanced version of the standard DQN that incorporates a priority-based action selection mechanism during the exploitation phase.
   • Logic: Before selecting the action with the highest Q-value, the algorithm checks sensor data. If the left sensor detects a closer obstacle, it prioritizes steering left; if the right sensor is closer, it prioritizes steering right. This heuristic guides the agent to avoid immediate collisions more aggressively than the raw Q-values alone.

3. Key Contributions

• Custom 2D Simulation: Development of a lightweight, custom Pygame environment tailored to a real-world map (University of Memphis) specifically for RL training.
• Sensor-Based State Representation: Implementation of a 7-sensor distance-based input system rather than raw pixel data, reducing computational complexity while maintaining spatial awareness.
• Priority-Based Action Selection: The primary novelty is the integration of a heuristic priority mechanism into the DQN's action selection process. This hybrid approach combines the learning capability of DQN with rule-based safety checks to improve navigation in tight spaces.
• Comparative Analysis: A rigorous comparison between Vanilla NN, Standard DQN, and the Modified DQN regarding convergence speed and final performance.

4. Results

The experiments were conducted on 1000 episodes using both CPU and GPU hardware.

Model	Average Reward	Training Time (Hours)	Performance Outcome
Original DQN	~25	10	Struggled to complete the full track; often got stuck or crashed on sharp turns.
Vanilla NN	~23	6	Could complete the track but required longer training time to stabilize.
Modified DQN	~40	4	Successfully completed the full track with the highest stability and speed.

• Performance Gain: The Modified DQN achieved an average reward approximately 60% higher than the Original DQN and 50% higher than the Vanilla Neural Network.
• Efficiency: The Modified DQN trained significantly faster (4 hours on GPU) compared to the Original DQN (10 hours), demonstrating that the priority mechanism accelerates convergence by reducing the exploration of unsafe states.

5. Significance and Future Work

• Significance: The paper demonstrates that integrating simple heuristic rules (sensor-based priority) with deep reinforcement learning can significantly outperform pure DQN in constrained 2D navigation tasks. It proves that for specific autonomous driving tasks, a hybrid approach can yield faster learning and safer policies than relying solely on neural network approximation.
• Limitations: The current study is limited to a 2D environment with constant speed and a single agent. The use of SUMO (Simulation of Urban MObility) for multi-vehicle traffic simulation was attempted but not fully realized due to complexity and time constraints.
• Future Directions:
  • Tuning hyperparameters (learning rate, network architecture).
  • Integrating the approach into the SUMO framework to simulate multi-vehicle traffic and complex urban scenarios.
  • Extending the model to handle multiple autonomous agents simultaneously.
  • Transitioning from simulation to real-world hardware implementation.