Technical Summary: OracleTSC

Problem Statement

Traffic Signal Control (TSC) systems require transparent decision-making to earn public trust, yet traditional Reinforcement Learning (RL) methods function as "black boxes." While Large Language Models (LLMs) offer a path to interpretability through natural language reasoning, applying them to TSC via reinforcement fine-tuning faces significant hurdles. Specifically, TSC is a long-horizon problem characterized by:

1. Sparse and Delayed Feedback: The impact of a signal change often manifests only after many timesteps, making credit assignment difficult.
2. Weak Reward Signals: Most feasible phase changes yield only marginal improvements in congestion metrics, resulting in noisy gradients that fail to guide policy optimization effectively.
3. Reasoning Drift: When LLM policies are uncertain, their generated explanations and phase choices vary widely across responses for the same state, injecting randomness into the control loop and causing performance instability.

Existing approaches that fine-tune LLMs for TSC often suffer from training instability and limited improvements over pre-trained baselines, largely because standard Proximal Policy Optimization (PPO) struggles to handle these long-horizon, weak-reward dynamics.

Methodology: OracleTSC

The authors propose OracleTSC, a framework designed to stabilize reinforcement fine-tuning for LLM-based TSC by addressing weak signals and reasoning instability. The approach formulates TSC as a Markov Decision Process (MDP) where states and actions are represented as natural language text, allowing the LLM to generate both a traffic signal phase and a Chain-of-Thought (CoT) explanation.

The core of the method involves two novel mechanisms integrated into the PPO training objective:

1. Reward Hurdle Mechanism (RHM)

To combat weak reward signals, OracleTSC introduces a Reward Hurdle. Instead of using raw environmental feedback (e.g., queue length reduction) directly, the method subtracts a calibrated threshold ($HR$) from the reward.

• Mechanism: $R_{total} = R_{env} - HR$.
• Effect: Actions that result in improvements smaller than the hurdle are penalized (receiving negative or zero reward), while only high-impact actions receive positive reinforcement. This filters out low-quality updates and forces the policy to focus on interventions that meaningfully reduce congestion, effectively acting as a constant baseline to reduce gradient variance in the early stages of training.

2. Uncertainty Regularization via Temperature-Scaled Softmax Discrete Semantic Entropy

To address reasoning drift and inconsistent decision-making, the framework introduces a regularization term based on semantic entropy.

• Mechanism: The model generates $G$ candidate responses for a given state. These responses are clustered by the extracted traffic phase (ignoring linguistic variations). A probability distribution over these phases is computed using a temperature-scaled Softmax.
• Reward Term: A penalty is applied if the distribution is high-entropy (i.e., the model is uncertain about the phase). The reward is defined as $R_{entropy} = p_{chosen} \times \mathbb{I}(R_{env} > HR)$, where $p_{chosen}$ is the probability of the selected phase.
• Effect: This encourages the model to produce consistent phase choices across multiple sampling attempts, stabilizing the reasoning process and reducing the variance in action selection.

The final objective combines the standard PPO clipped surrogate objective with these modified rewards, optimizing both the policy network and a value network.

Key Contributions

1. Identification of Failure Modes: The paper empirically identifies that insufficient suppression of suboptimal actions (due to weak rewards) and reasoning drift (due to output uncertainty) are primary causes of instability in LLM-based TSC.
2. Reward Hurdle Mechanism: A simple yet effective penalty that imposes an explicit performance threshold, amplifying learning signals from high-impact actions and suppressing low-quality updates.
3. Semantic Entropy Regularization: A principled approach to quantifying and mitigating uncertainty at the decision level (rather than the token level) using temperature-scaled softmax, which stabilizes long-horizon reasoning.
4. Demonstration of Robustness: The framework enables compact models (e.g., LLaMA3-8B, Qwen3-0.6B) to achieve substantial performance gains while preserving interpretability through natural language explanations.

Experimental Results

Experiments were conducted on the LibSignal benchmark using two distinct intersection topologies: CityFlow1×1 (Hangzhou, 8 phases) and Cologne1 (Germany, 4 phases).

• Performance Gains: OracleTSC significantly outperformed pre-trained LLM baselines and non-LLM RL methods in several metrics.
  • On CityFlow1×1, the fine-tuned LLaMA3-8B model achieved a 75% reduction in travel time and a 67% decrease in queue lengths compared to the pre-trained baseline.
  • On Cologne1, the method reduced queue lengths by 39–50% and travel times by 29–85%, achieving throughput comparable to state-of-the-art non-explainable RL controllers (e.g., PressLight, IDQN).
• Cross-Intersection Generalization: A policy trained on one intersection topology transferred effectively to a structurally distinct intersection without additional fine-tuning. For instance, a model trained on Cologne1 reduced travel time by 13% when tested on CityFlow1×1, and vice versa.
• Ablation Studies:
  • Using the environmental reward alone yielded marginal gains.
  • Adding the Reward Hurdle alone significantly improved performance by filtering weak signals.
  • Adding Uncertainty Regularization alone improved stability and reduced queue lengths.
  • Joint Application: The combination of both mechanisms yielded the best results, confirming their complementary nature.
• Qualitative Analysis: Post-fine-tuning, the LLM's CoT reasoning became structured, concise, and free of the self-contradictions and phase misidentifications observed in pre-trained models.

Significance and Claims

The paper claims that OracleTSC provides a new perspective for improving the effectiveness of reinforcement fine-tuning in long-horizon TSC tasks. Its significance lies in:

• Stabilizing LLM Control: Demonstrating that targeted reward shaping (hurdles) and uncertainty regularization can overcome the inherent instability of applying PPO to LLMs in sparse-reward environments.
• Bridging Interpretability and Performance: Achieving performance competitive with specialized, black-box RL controllers while maintaining the unique ability of LLMs to provide natural language rationales for decisions.
• Scalability: Showing that these mechanisms work across different model scales (from 0.6B to 8B parameters) and generalize across different intersection topologies, suggesting a pathway toward scalable, reusable, and interpretable intelligent transportation systems.

The authors modestly note that their goal is not to claim superiority over all existing optimized RL algorithms but to improve the reliability and transparency of LLM-based controllers. They acknowledge limitations, such as the need for manual tuning of the hurdle rate and the computational cost of multi-sample generation for entropy estimation.