Bridging the Sim-to-Real Gap in Reinforcement Learning-Based Industrial Dispatching through Execution Semantics

Technical Summary: Bridging the Sim-to-Real Gap in Industrial Dispatching via Execution Semantics

Problem Statement

The paper addresses the persistent "sim-to-real" gap in Reinforcement Learning (RL)-based industrial dispatching, specifically within Job-Shop Scheduling Problems (JSSP). While RL has shown promise in simulated environments, its industrial adoption is hindered by a structural mismatch between policy assumptions and execution reality.

Current literature primarily attempts to bridge this gap by increasing policy robustness (e.g., training under disturbances, imitation learning, meta-learning). However, these approaches retain a critical assumption: that a valid, synchronized decision state and a consistent action interface are available at inference time. In industrial settings, this is rarely true due to:

1. Asynchronous Observations: Decisions are made on reconstructed states derived from heterogeneous, delayed event streams (MES, ERP, physical sensors), leading to partially observable environments where decisions valid in the observation may be infeasible in reality.
2. Undefined Execution Semantics: There is no explicit definition of action admissibility or how decisions translate into system actions across heterogeneous systems.
3. Ambiguous Outcome Attribution: Execution failures are recorded as generic disturbances, conflating policy errors with physical constraints, transactional inconsistencies, or human interventions. This lack of differentiation prevents structured evaluation and policy refinement.

The authors argue that the core limitation is not merely policy quality, but the absence of a mediating layer that explicitly manages execution semantics.

Methodology

The paper proposes a policy-neutral execution and measurement layer that acts as middleware between dispatching policies and the industrial execution environment. This layer does not design the scheduling policy but governs how decisions are formed, validated, and recorded. The architecture consists of three core mechanisms:

1. State Snapshot Isolation:

   • To handle asynchronous data, the layer accumulates physical and transactional events in an execution cache.
   • Upon a dispatch trigger, it isolates a point-in-time snapshot ($s_k$) representing the system state. This snapshot is immutable during the decision process to prevent temporal inconsistencies.
   • Pre-commit Invalidation: If critical updates arrive before an action is committed, the decision is aborted, the snapshot discarded, and a new isolation step initiated. This prevents the execution of decisions based on stale information.

2. Policy-Neutral Execution Contract:

   • The layer defines a standardized decision request $R_k = \langle s_k, m_k, J^{cand}_k \rangle$, where $J^{cand}_k$ is the candidate set.
   • The candidate set is the intersection of physically feasible actions (shop floor reality) and transactionally admissible actions (MES/ERP constraints).
   • This ensures the policy only receives actions that are valid under both physical and transactional constraints, explicitly defining action admissibility.

3. Divergence Representation and Outcome Attribution:

   • Instead of a single outcome signal, the layer records a divergence tuple $D_k = \langle a^{intent}_k, x^{sys}_k, x^{phys}_k, u_k \rangle$.
   • This tuple separates:
     • $a^{intent}_k$: The policy's intended action.
     • $x^{sys}_k$: The transactional system response (e.g., MES rejection).
     • $x^{phys}_k$: The actual physical execution outcome.
     • $u_k$: Human interventions/overrides.
   • This structure distinguishes between visible divergence (policy error due to observable conflicts) and hidden divergence (errors due to unobserved system states), enabling structured attribution of execution failures.

Empirical Evaluation

The framework was evaluated using a discrete-event simulation (SimPy) with a single non-preemptive machine. The study isolated the effects of the execution layer from the dispatching logic by testing standard heuristics (Earliest Due Date and Shortest Processing Time) under three observation lag regimes (Low, Medium, High).

Key Results:

• Decision Validity: Under medium observation lag, the execution layer reduced invalid dispatches from 61.81% (direct execution) to 48.68%.
• Operational Performance: The layer significantly reduced mean weighted tardiness at low (1.08 vs. baseline) and medium (6.43 vs. 11.21) lag regimes. At high lag, where updates arrive after the decision window, the operational benefit diminished, converging toward direct execution performance.
• Action Admissibility: The execution contract reduced "visible divergence" (actions committed despite known conflicts) to 0.00, whereas direct execution exhibited 15.90% visible divergence.
• Throughput: No measurable throughput penalty was observed, indicating the layer constrains when actions are committed without altering system capacity.
• Attribution Coverage: Unlike direct execution, which collapses failures into generic signals, the proposed layer achieved 100% attribution coverage, categorizing disturbances into transactional, physical, combined, and human-override types.

Key Contributions and Significance

The paper claims its primary contribution is methodological and architectural, shifting the focus from policy-level robustness to explicit execution semantics at deployment time.

1. Structural Diagnosis: It identifies that the sim-to-real gap is exacerbated by the lack of decision-valid state representations and explicit action admissibility definitions, rather than just policy suboptimality.
2. Execution Semantics: The proposed layer provides a standardized interface that makes deployment mismatches observable, reusable, and comparable. It transforms execution uncertainty into supervisory data.
3. Attribution and Learning: By separating policy intent from execution outcomes, the layer enables the distinction between policy errors and environmental limitations. This structured data is essential for downstream tasks such as offline reinforcement learning, disturbance-aware value estimation, and the systematic reuse of human interventions.

The authors conclude that while the operational benefits (reduced tardiness) are most pronounced when updates arrive within the decision window, the analytical benefits (structured attribution and interpretability) persist across all lag regimes. This makes the layer a prerequisite for deployable, event-driven industrial dispatching systems, facilitating the transition from "black box" execution to interpretable, data-driven policy refinement.