🤖 machine learning

DGLight: DQN-Guided GRPO Fine-Tuning of Large Language Models for Traffic Signal Control

The paper introduces DGLight, a novel framework that fine-tunes a pretrained large language model for traffic signal control by leveraging a Deep Q-Network critic to guide Group Relative Policy Optimization, resulting in a system that achieves state-of-the-art performance among LLM-based controllers while providing interpretable reasoning traces.

Original authors: Chenbo Yu

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenbo Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy city intersection as a chaotic dance floor. Cars are dancers trying to move in and out, but if everyone tries to dance at once, they bump into each other, creating a traffic jam. The "Traffic Signal Controller" is the DJ who decides who gets to dance (go) and who has to wait (stop).

For a long time, DJs used two main methods:

The Metronome DJ (Fixed-Time): They play the same beat every 30 seconds, regardless of how many people are on the floor. It's simple, but if a huge crowd arrives, they can't adapt.
The Experienced Human DJ (Traditional RL): This DJ watches the crowd and learns from mistakes. They get better over time, but they are often "black boxes." You can't ask them why they chose a specific beat, and if you move them to a different club (a different city), they might get confused because they learned the specific vibe of the first club.

Enter DGLight: The "Smart Intern" with a "Veteran Coach"

This paper introduces DGLight, a new way to train a Traffic Signal Controller using a Large Language Model (LLM)—basically, a super-smart AI that is great at writing and reasoning.

Here is how DGLight works, using a simple analogy:

1. The Student (The LLM)

Think of the LLM as a brilliant intern who can write a detailed report explaining why they made a decision. "I see the East lane is clogged, so I'll let East traffic go." This is great because, unlike the "black box" DJ, you can read the intern's notes and understand their logic. However, this intern is new to traffic; they know how to write but don't know how to drive a city.

2. The Coach (The DQN Critic)

To teach the intern, the researchers hired a "Veteran Coach." This coach is a traditional AI (a Deep Q-Network based on a method called CoLight) that has spent years watching traffic in a specific city (Hangzhou). The coach is an expert at knowing exactly which signal phase will reduce congestion right now.

3. The Training Camp (The Two-Stage Process)

The magic of DGLight happens in two stages:

Stage 1: The Coach Gets Certified. First, the Veteran Coach is trained on real traffic data until it becomes an expert at predicting which signal choice is best.
Stage 2: The Intern Learns from the Coach. Now, the Intern (LLM) starts practicing.
- The Intern looks at a traffic situation and writes a plan: "I think we should let North traffic go because..."
- Instead of waiting to see if the traffic actually clears up (which takes a long time and is slow to learn), the Coach immediately looks at the Intern's plan and gives it a score. "Good plan, +10 points!" or "Bad plan, -5 points."
- The Intern uses these scores to learn. Because the Coach gives a score for every possible move (not just the one that happened), the Intern learns much faster and more deeply.

Why is this special?

1. It Speaks Human (Interpretability)
Most traffic AI just outputs a number (e.g., "Switch to Phase 2"). DGLight outputs a sentence: "I am switching to Phase 2 because the East lane has a long queue." It's like having a traffic controller who explains their reasoning in plain English.

2. It's a Smart Generalist (Transferability)
Usually, if you train a traffic AI in City A, it fails in City B. But because DGLight learns the logic of traffic (how to reason about queues and delays) rather than just memorizing the streets of City A, it works surprisingly well in completely new cities (like Jinan) that the Coach never even saw. It's like an intern who learns the principles of crowd control, so they can manage a party in a new city without needing to relearn everything.

3. It's Better Than the Old Guard
The paper tested DGLight against the old "Metronome" methods and the "Experienced Human DJ" methods.

Result: DGLight was the best among all the AI methods that use language.
Result: It was just as good as, and sometimes better than, the strongest traditional traffic AI, but with the added bonus of being able to explain its decisions.

What did they not do?

The paper is very careful to say what this is not.

It is not a magic wand that solves all traffic forever.
It is not being tested on real-world live traffic yet (only in simulations).
The "Coach" was trained on one city, but the "Intern" learned to generalize. The paper notes that while the Intern is good at new cities, it's not perfect, and the system still relies on the Coach's initial training.

The Bottom Line

DGLight is a system that teaches a smart, talking AI how to control traffic lights by having it practice under the supervision of a traditional traffic expert. The result is a controller that is as smart as the experts but can also tell you why it made its choices, making it a more trustworthy and adaptable tool for managing city traffic.

1. Problem Statement

Traffic Signal Control (TSC) is critical for reducing urban congestion. While traditional methods (Fixed-Time, Max-Pressure) lack adaptability and Reinforcement Learning (RL) methods (e.g., CoLight, MPLight) struggle with generalization and interpretability, Large Language Models (LLMs) offer a promising alternative due to their reasoning capabilities and ability to generate interpretable text.

However, applying LLMs to TSC faces two main challenges:

Domain Adaptation: Standard LLMs lack specific traffic dynamics knowledge, leading to suboptimal decisions.
Reward Sparsity & Interpretability: Traditional RL relies on sparse, cumulative environment rewards (e.g., total delay reduction over an episode), which are difficult for LLMs to optimize directly. Furthermore, end-to-end LLM fine-tuning often lacks the dense, state-specific supervision needed to learn complex traffic policies.

The paper proposes DGLight, a framework that bridges these gaps by using a value-based critic to guide the fine-tuning of an LLM, enabling it to learn from dense, traffic-aware action values while producing interpretable reasoning.

2. Methodology: The DGLight Framework

DGLight employs a two-stage reinforcement learning pipeline that combines a Deep Q-Network (DQN) critic with Group Relative Policy Optimization (GRPO).

Stage 1: Training the CoLight-Based DQN Critic

Objective: Train a value estimator to provide dense, traffic-aware action values for signal phase decisions.
Architecture: Based on CoLight, a graph-attention model that captures network-level cooperation between intersections.
Process:
- The critic observes structured traffic states (lane queue lengths, delays).
- It outputs a Q-value vector ( $Q(s, a)$ ) for all admissible signal phases at each intersection.
- It is trained using standard Bellman error minimization with experience replay.
Role: This critic acts as a "frozen" teacher that evaluates the quality of any potential signal action based on complex spatial and temporal traffic contexts, providing a richer signal than raw environment rewards.

Stage 2: LLM Fine-Tuning with GRPO

Input: The LLM (based on Llama3-8b) receives a textual prompt summarizing the traffic state, task description, and available signal phases.
Sampling: For each prompt, the policy samples a group of $k$ candidate responses (reasoning traces + signal decisions).
Parsing & Scoring:
- Each response is parsed to extract the proposed signal phase.
- The frozen DQN critic scores each candidate action using its pre-trained Q-values ( $Q(s, a)$ ).
- Invalid actions receive a fixed fallback reward.
Optimization (GRPO):
- Instead of using a separate value model (as in PPO), DGLight uses Group Relative Policy Optimization (GRPO).
- Advantages are computed by comparing the critic scores of the $k$ sampled responses within the same group.
- The policy is updated to maximize the relative advantage, effectively transferring the critic's traffic knowledge into the LLM's reasoning and decision-making process.

3. Key Contributions

Unified Framework: Proposes DGLight, the first framework to unify a value-based traffic critic (CoLight-DQN) with LLM fine-tuning via GRPO. It allows LLMs to learn from dense, per-state action values rather than sparse cumulative rewards.
Interpretable Policy: The system trains the LLM to output both a valid signal decision and a human-readable reasoning trace (e.g., analyzing queue lengths and neighbor traffic), bridging the gap between performance and explainability.
Empirical Validation: Conducts comprehensive experiments on standard benchmarks (Jinan and Hangzhou datasets), demonstrating that DGLight outperforms existing LLM-based controllers and remains competitive with state-of-the-art RL baselines.
Transferability: Shows that the method generalizes well to unseen city datasets (e.g., training on Hangzhou-1 and testing on Jinan), suggesting the LLM learns generalizable traffic reasoning patterns rather than overfitting to specific city data.

4. Experimental Results

The study evaluated DGLight against:

Traditional Methods: FixedTime, MaxPressure.
RL Baselines: MPLight, CoLight, PressLight, etc.
LLM Baselines: LightGPT, JS-GRPO variants.

Key Findings:

Performance: DGLight achieved the best overall performance among all LLM-based controllers across most metrics (Average Traveling Time, Average Queue Length, Average Waiting Time).
Comparison with RL: DGLight remained competitive with strong RL baselines (like CoLight and Advanced-CoLight) and even outperformed them on certain metrics, despite the added complexity of reasoning generation.
Transferability: DGLight showed superior transferability to unseen cities compared to both RL baselines and other LLM methods, indicating robust generalization.
Ablation Study:
- Moving from a base LLM (Llama3-8b) to a traffic-adapted model (LightGPT) yielded the largest improvement.
- The addition of Critic-Guided GRPO provided further significant gains, confirming the value of the dense reward signal.
Interpretability: Qualitative analysis confirmed that the generated reasoning traces were logically consistent with the chosen signal phases (e.g., prioritizing lanes with the longest queues).

5. Significance and Future Work

Significance:

Bridging the Gap: DGLight successfully demonstrates that LLMs can be effectively adapted for complex control tasks like TSC without sacrificing the interpretability that makes them unique.
Efficient Training: By using a frozen critic to generate dense rewards, the method avoids the instability of training LLMs with sparse environment rewards.
Real-World Applicability: The interpretability of the model's decisions addresses a critical barrier to the real-world adoption of AI in traffic management, where accountability and explainability are paramount.

Future Directions:

Token-Level Reward Assignment: Refining the reward signal to distinguish between formatting tokens and critical reasoning/decision tokens to improve training efficiency.
MLLM-Style Architecture: Moving beyond text-serialized states to a Multi-Modal LLM (MLLM) approach with a dedicated traffic situation encoder to better capture spatial and numerical relationships in the road network.

In conclusion, DGLight represents a significant step forward in applying Generative AI to urban infrastructure, proving that LLMs can learn high-performance, interpretable control policies when guided by robust value-based critics.