⚡ electrical engineering

Event-Driven Reinforcement Learning Enables Long-Horizon Control in Semiconductor Fabrication

This paper proposes a scalable, event-driven deep reinforcement learning framework that effectively optimizes long-horizon, multi-objective control in complex semiconductor fabrication systems, demonstrating significant improvements in throughput and utilization across diverse industry-real scenarios.

Original authors: Yavar Yeganeh, Mahsa Shekari, Nicla Frigerio, Daniele Pagano, Andrea Matta

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yavar Yeganeh, Mahsa Shekari, Nicla Frigerio, Daniele Pagano, Andrea Matta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a semiconductor factory (a "fab") as a massive, chaotic kitchen where thousands of different recipes are being cooked simultaneously. The ingredients are silicon wafers, and the chefs are hundreds of different machines. The problem is that the kitchen is incredibly complex: some recipes take hours, some machines break down, some ingredients need to be kept in a specific order, and the "chefs" (machines) often have to wait for each other.

Traditionally, factory managers use simple rules to decide which wafer goes to which machine next (like "First In, First Out" or "Shortest Job First"). But in this chaotic kitchen, simple rules often lead to traffic jams, wasted time, and slow cooking.

This paper proposes a new way to run the kitchen using Artificial Intelligence (AI) that learns by trial and error, a method called Reinforcement Learning (RL). Here is the breakdown of their approach and what they found:

The Core Problem: The "Long Wait"

In a normal video game, you press a button, and you get points immediately. In a semiconductor factory, if you make a decision today (like sending a wafer to Machine A), you might not see the result (like a finished product) for days. The reward is delayed.

The Analogy: Imagine you are playing a game of chess, but you only get a score at the very end of the match. If you make a bad move in the first minute, you won't know it until the game is over. Standard AI struggles with this because it doesn't know which specific move caused the win or loss.

The Solution: An "Event-Driven" Conductor

The researchers built a framework where the AI acts like a centralized conductor for the entire orchestra of machines, rather than just a soloist.

Thinking in "Events," Not Seconds: Instead of checking the factory every second, the AI only wakes up when something happens (an "event"), like a machine finishing a job or becoming free. This matches how the factory actually works.
Grouping the Chaos: Because the results are delayed, the AI doesn't judge a single decision in isolation. Instead, it looks at a group of events that happened over a chunk of time (like a 6-hour shift). It asks: "Did this whole group of decisions lead to a better kitchen overall?"
The Reward System: The AI gets two types of feedback:
- Immediate feedback: "You made Machine A wait 5 minutes; that's a small penalty."
- Big-picture feedback: "Over the last 6 hours, we produced 20% more wafers; that's a huge reward."
  By combining these, the AI learns to make small decisions that lead to big wins later.

How They Tested It

They didn't just guess; they built a digital twin (a super-accurate video game simulation) of a real-world semiconductor factory. They tested their AI in two ways:

Offline Learning: The AI studied a library of past factory logs (like reading a history book) without touching the real machines. This is safe because the AI can't accidentally break anything while learning.
Online Learning: The AI then practiced in the simulation, making real-time decisions and learning from the results, similar to a pilot training in a flight simulator.

The Results: A Smoother Kitchen

When they compared their AI to the standard rules used in factories today:

Throughput (Cooking Speed): The AI increased the number of finished wafers by up to 39% compared to a random approach and significantly beat standard rules.
Utilization (Chef Efficiency): The machines stayed busy and productive much more often, reducing idle time.
Consistency: The AI performed well across 31 different factory scenarios, showing it wasn't just lucky in one specific situation.

Why This Matters

The paper claims that by changing how the AI learns (focusing on groups of events and delayed rewards) rather than just changing the AI's brain, they can solve complex, long-term problems in massive systems.

In short: They taught an AI to be a master conductor for a chaotic factory orchestra. By listening to the whole symphony over time rather than just one note, the AI learned to keep the music playing smoothly, resulting in more products being made faster and with less waste.

Technical Summary: Event-Driven Reinforcement Learning for Long-Horizon Control in Semiconductor Fabrication

1. Problem Statement

Semiconductor manufacturing facilities (fabs) represent complex adaptive systems characterized by stochastic processing times, re-entrant process flows, heterogeneous equipment, and strict technological constraints. Decision-making in this domain, particularly at the dispatching level, involves sequential choices where the consequences of an action are often delayed and entangled with thousands of subsequent interactions.

Traditional reinforcement learning (RL) approaches face significant challenges in this setting:

Long-Horizon Credit Assignment: Standard one-step transition models misalign temporal credit assignment because system-level Key Performance Indicators (KPIs) like throughput and utilization are cumulative outcomes of many interdependent decisions, not immediate consequences of single actions.
Delayed Feedback: Rewards are sparse and observed only over long time segments, making learning unstable.
Scalability and Generalization: Many existing studies simplify the problem to localized control or rely on small-scale testbeds, failing to address system-wide policies under real industrial complexity. Furthermore, there is a lack of systematic comparison across algorithms regarding generalization and transferability in offline and online regimes.

2. Methodology

The authors propose a modular, event-driven deep reinforcement learning framework designed for centralized policy optimization in continuous, infinite-horizon environments.

Core Architecture

Centralized Agent: A single agent controls the entire system at the dispatching level, selecting candidate wafer lots for available equipment. This approach utilizes parameter sharing to improve scalability compared to multi-agent decompositions that require explicit coordination.
Event-Driven Dynamics: The system is modeled as a sequence of discrete events (e.g., equipment becoming idle) rather than fixed time steps. Actions trigger temporally extended, overlapping events.
State and Action Encoding:
- State: Encodes workload statistics, equipment status, and WIP distribution across process-flow sectors and product categories (fixed-size representation >5000 dimensions).
- Action: Encodes candidate lot features (product type, processing time, constraints). The policy uses a score-based softmax mechanism to handle variable action set sizes (up to thousands of candidates).
Constrained Policy: A hard inductive bias enforces domain-specific constraints (e.g., corrosion avoidance) by masking infeasible actions, while the reward function encourages constraint-compliant behavior.

Reward Construction

To address feedback sparsity, the reward is decomposed into two components:

Event-Level Reward ( $r_e$ ): A dense, shaped reward providing immediate local feedback based on processing time, idle time, and constraint violations.
System-Level Reward ( $r_g$ ): A delayed reward computed over a finite time window ( $t_s$ ) based on KPIs such as cumulative wafer moves (production rate) and average equipment utilization.
The total reward is $r = r_e + r_g$ .

Event-Group Temporal-Difference (TD) Learning

The paper introduces a novel TD formulation that respects the temporal structure of overlapping events:

Group Aggregation: Instead of updating each event independently, the framework samples a time segment $I_h$ and treats all events within it as a single group $G$ .
Group-Wise Objective: The learning objective minimizes the discrepancy between the aggregated event-level TD errors and the system-level reward for that segment. This allows global reward signals to propagate end-to-end across interdependent events.
Aggregation Strategies: The authors compare mean aggregation (smoothing) and weighted aggregation (accounting for event duration) to stabilize optimization.

Algorithmic Integration

The framework is algorithm-agnostic, integrating with various model-free backbones:

Offline Training: Utilizes historical data (random policy) with conservative methods (CQL, IQL) to mitigate distributional shift and overestimation.
Online Training: Employs off-policy (DQL, SAC) and on-policy (PPO) methods for fine-tuning. SAC utilizes entropy regularization for exploration, while PPO uses a group-relative surrogate advantage to align with the grouped TD structure.

3. Key Contributions

Modular Event-Driven RL Framework: A scalable architecture for policy interaction and optimization in complex adaptive systems, using semiconductor fabrication as a representative real-world case.
Centralized Formulation: A unified policy-optimization problem over event-driven dynamics, enabling system-wide decision-making without the complexity of multi-agent coordination.
Multi-Phase Optimization: Tailored formulations supporting both offline pre-training (data-efficient, conservative) and online fine-tuning (adaptive, constrained exploration).
Generalization-Focused Evaluation: A comprehensive assessment across 31 diverse industrial scenarios, clarifying the relative strengths of different RL formulations and algorithms in long-horizon settings.

4. Experimental Results

Experiments were conducted using a high-fidelity simulator based on real industrial data from STMicroelectronics, covering 31 scenarios with varying Work-in-Progress (WIP) and operational configurations.

Baselines: Performance was compared against First-In-First-Out (FIFO), Shortest Processing Time (SPT), and a Random policy.
Offline Performance: Agents trained on random-policy data achieved significant gains over the Random baseline (up to ~39% throughput, ~28% saturation) and outperformed FIFO. Among offline algorithms, Double Q-Learning (DQL) and Conservative Q-Learning (CQL) variants showed robust performance.
Online Performance: Online fine-tuning further improved performance, achieving up to ~42% throughput and ~30% saturation gains over the no-control (Random) setting. Soft Actor-Critic (SAC) with high target entropy and DQL demonstrated strong sample efficiency.
Ablation Studies:
- TD Formulation: The proposed event-aggregated TD objective significantly outperformed standard truncated discounted sequences and stacked event-wise objectives, confirming the importance of group-level credit assignment.
- Reward Design: Training with system-level segment rewards alone proved effective, though combining event-level shaping accelerated convergence in some cases.
- Algorithm Comparison: Offline methods required conservatism (CQL/IQL) to avoid overestimation, while online methods benefited from entropy regularization (SAC) and stable policy gradients (PPO).

5. Significance and Claims

The paper claims that the proposed framework successfully addresses the core challenges of applying RL to large-scale, event-driven systems with delayed feedback. By decoupling environment interaction from policy optimization and introducing a tailored temporal-difference layer, the framework enables agents to learn system-wide policies that optimize multiple KPIs simultaneously.

The authors emphasize that:

Scalability: The centralized, feature-based approach scales to high-dimensional settings without the coordination overhead of multi-agent systems.
Transferability: The framework demonstrates consistent performance gains across diverse operating scenarios, suggesting robustness to changes in WIP and product mixes.
Practicality: The ability to pre-train offline on historical data and fine-tune online with limited interaction makes the approach viable for industrial deployment where exploration is costly and risky.

The study concludes that careful problem formulation—specifically the alignment of temporal structure with reward propagation—is as critical as the choice of the underlying RL algorithm for achieving effective control in complex adaptive systems.