C$^2$T: Captioning-Structure and LLM-Aligned Common-Sense Reward Learning for Traffic--Vehicle Coordination

1. Problem Statement

Current state-of-the-art urban traffic control systems rely heavily on Multi-Agent Reinforcement Learning (MARL) to coordinate Traffic Light Controllers (TLCs) and Connected Autonomous Vehicles (CAVs). However, these systems face a fundamental limitation:

• Myopic Rewards: They utilize hand-crafted, local reward functions (e.g., minimizing queue length, intersection pressure, or average delay).
• Lack of Human Alignment: These metrics fail to capture high-level, human-centric goals such as safety, flow stability, and passenger comfort.
• Negative Consequences: Optimizing solely for local efficiency often leads to global instability, including congestion waves, unstable traffic flows, frequent harsh braking, and unsafe vehicle spacing, even when average travel times appear optimal.

The core challenge is to learn a human-aligned intrinsic reward that reflects "common sense" traffic quality (balancing efficiency with safety and comfort) without requiring online interaction with human experts or incurring the latency of real-time Large Language Model (LLM) calls.

2. Methodology: The C2T Framework

The authors propose C2T, a three-stage framework that distills common-sense knowledge from an LLM into a lightweight, learnable reward function.

Stage 1: Structured Captioning and Preference Pairing

Instead of feeding raw sensor data or free-form text to an LLM, C2T converts simulator observations into deterministic, unit-aware structured captions.

• Schema: The caption includes specific fields: signal phase, elapsed time, queue lengths, pressure, average delay, throughput, safety metrics (Time-to-Collision percentiles, harsh braking counts, red-light risk flags), and nearest vehicle kinematics.
• Pair Sampling: The system samples pairs of observations $(o_a, o_b)$ that exhibit high contrast in congestion, safety, or short time shifts.
• LLM Judgment: A fixed prompt queries the LLM to decide which caption represents a "better" traffic state. Only decisive labels (1 or 2) are retained to build a preference dataset $D_{pref}$.

Stage 2: Offline Preference Learning

A lightweight neural scorer $r_\phi$ is trained offline to mimic the LLM's preferences.

• Architecture: The scorer takes the structured caption tokens and auxiliary numeric values as input and outputs a scalar quality score.
• Loss Function: The model is trained using a Bradley-Terry loss on the preference pairs, weighted by the frequency of the template to reduce bias.
• Output: The result is an intrinsic reward function $r_\phi$ that predicts the "common-sense" quality of a state without needing the LLM during training.

Stage 3: Asymmetric Reward Integration & Safety Masking

The learned intrinsic reward is integrated into the standard MARL training loop (specifically PPO) for Traffic Light Controllers.

• Asymmetric Mixing: The intrinsic reward is applied only to TLC agents, while vehicles (in this instantiation) remain driven by standard environment dynamics. This avoids non-stationarity issues that would arise if both agents and vehicles adapted to a changing intrinsic signal simultaneously.
• Safety Mask: A critical component is a risk mask $m(o)$. If the state indicates high risk (e.g., low Time-to-Collision, harsh deceleration clusters, or red-light violations), the intrinsic reward is suppressed ($m=0$). This ensures the "common sense" reward never overrides immediate safety constraints.
• Normalization & Scheduling:
  • External rewards, intrinsic rewards, and safety penalties are treated as separate streams, normalized (z-score), and softly clipped to prevent dominance by any single signal.
  • A curriculum schedule $\lambda(t)$ gradually increases the weight of the intrinsic reward, allowing the policy to first satisfy basic environment constraints before optimizing for common-sense preferences.

3. Key Contributions

1. Structured Captioning for Traffic RL: A novel, deterministic schema that translates complex traffic states into unit-aware text, making them comparable by LLMs and reproducible across runs.
2. Offline Preference Learning: A method to distill LLM judgments into a reusable, lightweight intrinsic reward scorer, eliminating the need for online LLM calls during RL training.
3. Safe, Asymmetric Integration: A robust mechanism to inject human-aligned guidance into multi-intersection RL via asymmetric mixing (TLCs only), risk masking, and per-stream normalization, ensuring compatibility with standard PPO.
4. Flexibility: The framework allows for distinct policy behaviors (e.g., "efficiency-focused" vs. "safety-focused") simply by modifying the LLM prompt used during the offline data collection phase.

4. Experimental Results

The framework was evaluated on CityFlow benchmarks using three real-world networks: Jinan, Hangzhou, and New York (196 intersections), under various stress tests (extreme demand surges, 24-hour cycles).

• Performance Gains: C2T significantly outperformed strong MARL baselines (e.g., MPLight, CoLight, AttendLight) and LLM-based agents (LLMLight).
  • Efficiency: Reduced Average Travel Time (ATT) and Average Queue Length (AQL).
  • Safety: Increased Time-to-Collision (TTC) percentiles and reduced harsh braking events.
  • Energy: Improved energy-related proxies (fewer stops, lower speed variance).
• Generalization: Models trained on smaller networks (Jinan) successfully transferred to larger networks (New York) and different cities (Hangzhou) with zero-shot or few-shot adaptation.
• Ablation Studies:
  • Removing the intrinsic reward collapsed performance to baseline levels.
  • Removing the safety mask improved efficiency slightly but drastically degraded safety metrics.
  • Removing per-stream normalization caused training instability.
  • Using a curriculum schedule was essential for stable convergence.

5. Significance

This work represents a paradigm shift in traffic control optimization:

• From Hand-Crafted to Learned Rewards: It moves away from brittle, manually designed metrics toward rewards learned from high-level human (or AI-simulated human) intuition.
• Bridging the Gap: It successfully bridges the gap between the interpretability of LLMs and the efficiency of RL, solving the latency and scalability issues of using LLMs directly in control loops.
• Safety-Centric AI: By explicitly incorporating a risk mask, the framework ensures that "common sense" optimization never compromises immediate safety, addressing a critical failure mode in previous RL traffic systems.
• Scalability: The offline nature of the reward learning makes the approach highly scalable to large, complex urban networks without increasing online computational costs.

In summary, C2T demonstrates that integrating structured, LLM-derived common sense into MARL can produce traffic control policies that are not only more efficient but also safer, smoother, and more aligned with human expectations.