⚡ electrical engineering

Addressing Terminal Constraints in Data-Driven Demand Response Scheduling

This paper proposes a hybrid reinforcement learning approach that integrates Goal-Space Planning with Deep Deterministic Policy Gradient to enhance sample efficiency and satisfy long-horizon terminal constraints in data-driven demand response scheduling for electrified chemical processes.

Original authors: Maximilian Bloor, Martha White, Ehecatl Antonio del Rio Chanona, Calvin Tsay

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maximilian Bloor, Martha White, Ehecatl Antonio del Rio Chanona, Calvin Tsay

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Flexible Factory" Problem

Imagine a giant factory that makes nitrogen gas (like an Air Separation Unit, or ASU). This factory is connected to the power grid, which is like a giant, chaotic marketplace for electricity. Sometimes electricity is cheap (like a sale at the grocery store), and sometimes it is incredibly expensive (like buying water in a desert).

The factory wants to be smart: it wants to run its machines hard when electricity is cheap and slow down when it's expensive. This is called Demand Response.

However, there's a catch. The factory has a giant storage tank for its product.

If it runs too hard when electricity is cheap, the tank overflows.
If it slows down too much when electricity is expensive, the tank runs dry.

The most critical rule is the Terminal Constraint: At the very end of the day (or the scheduling period), the tank must have a specific amount of product left in it. If the tank is empty at the end, the factory crashes the next morning because it has nothing to sell.

The Problem: The "Short-Sighted" Student

The researchers tried to teach a computer (using an AI technique called Reinforcement Learning) how to manage this factory. They wanted the AI to learn the perfect schedule to save money while keeping the tank full enough at the end.

But the AI kept failing. It was like a student who only studies for the test happening right now.

The AI's Mistake: When electricity prices dropped, the AI would say, "Great! Let's make as much product as possible right now!" It would fill the tank. But then, when prices spiked later, it would panic and stop production. By the time the day ended, the tank was empty.
Why? The AI couldn't connect the dots. It didn't understand that the decision it made at 8:00 AM (filling the tank) would cause a disaster at 8:00 PM (running out of product). In AI terms, this is called a long-horizon credit assignment problem. The "reward" (or punishment) for a mistake happens too far in the future for the AI to learn from it.

The Solution: The "Map of Goals"

The authors fixed this by giving the AI a new way of thinking. Instead of just looking at the immediate next step, they introduced a system called Goal-Space Planning (GSP).

Think of it like this:

The Old Way (Standard AI): The AI is walking through a dark forest, taking one step at a time. It only knows if it tripped right now. It has no idea that a cliff is 100 steps away, so it keeps walking toward it.
The New Way (GSP): The researchers gave the AI a map with checkpoints.
- They divided the day into time blocks (e.g., 8 AM–12 PM, 12 PM–4 PM).
- They divided the tank levels into zones (e.g., "Low," "Medium," "High").
- They created a "Goal Map" that says: "If you are at 'Low Tank' at 8 AM, you MUST reach 'Medium Tank' by 12 PM to have a chance of surviving until 8 PM."

The AI learns to plan its moves by jumping from one checkpoint to the next on this map, rather than just looking at the next second. It learns that "High Tank at 8 AM" is a valuable goal because it leads to "Safe Tank at 8 PM."

How It Works in Practice

The researchers tested this on a simulated version of the nitrogen factory. They compared three things:

Standard AI (DDPG): The short-sighted student.
Offline GSP: The student who studied the map before starting the test.
Online GSP: The student who learns the map while taking the test.

The Results:

Speed: The standard AI took a very long time to learn, and even then, it often failed to keep the tank full at the end. The GSP versions learned much faster (about 50% faster in terms of training steps).
Success: The GSP agents successfully kept the tank at the right level at the end of the day. They learned to "save" product during cheap times to ensure they had enough buffer for expensive times, all while keeping the final tank level safe.
The "Myopic" Fix: The standard AI kept draining the tank to save money right now. The GSP AI understood that saving a little bit of product now was worth it to avoid a disaster later.

The Takeaway

The paper shows that by breaking a long, complicated problem into smaller, manageable "goals" (like checking the tank level at specific times), we can teach AI to think long-term.

Instead of just reacting to the price of electricity second-by-second, the AI now plans its day like a chess player, looking several moves ahead to ensure it doesn't lose the game (run out of product) at the very end. This makes the factory more flexible, saves money, and keeps the system stable without needing complex physics equations to be programmed by hand.

Technical Summary: Addressing Terminal Constraints in Data-Driven Demand Response Scheduling

Problem Statement
Electrified chemical processes, such as Air Separation Units (ASUs), are increasingly incentivized to participate in Demand Response (DR) schemes to leverage time-varying electricity markets. This requires operating flexibly—reducing production during high-price periods and utilizing stored inventory to meet demand—while adhering to strict operational constraints. A critical challenge in this domain is satisfying terminal constraints (e.g., ensuring a minimum level of product storage at the end of a scheduling horizon) to maintain dynamic stability for subsequent periods.

Conventional model-based optimization methods (e.g., Nonlinear MPC) are computationally expensive and difficult to maintain due to the complexity of first-principle models. While Reinforcement Learning (RL) offers a data-driven alternative with low online computational cost, model-free RL algorithms struggle with long-horizon credit assignment. Specifically, when terminal constraints are enforced via sparse, delayed reward signals (e.g., a penalty only at the final time step), the agent fails to link early control actions to distant outcomes. This often leads to myopic behavior, where the agent depletes storage to minimize immediate costs, violating terminal constraints and destabilizing the process.

Methodology
The authors propose a hybrid framework integrating Goal-Space Planning (GSP) with the Deep Deterministic Policy Gradient (DDPG) algorithm to address long-horizon credit assignment. The approach is built upon a hierarchical control structure where a high-level RL agent computes optimal setpoints, and a low-level Linear Model Predictive Controller (LMPC) tracks these setpoints.

The core methodological contributions include:

Goal Formulation: Instead of treating terminal constraints solely as constrained optimization problems, the authors encode them as desirable goal states. For the ASU, the goal space ( $G$ ) is defined by discretizing the product storage level and time periods. This abstraction reduces the high-dimensional state space (including impurities and temperatures) into a low-dimensional representation focused on the constraint variable (storage).
Temporally Abstract Models: The system learns two types of models:
- State-to-Goal Models: Neural networks ( $r_\gamma, \Gamma$ ) that estimate the discounted return and discount factor from a current state to a reachable subgoal.
- Goal-to-Goal Models: Learned transitions ( $\tilde{r}, \tilde{\Gamma}$ ) between adjacent time periods in the goal space, constructed from observed trajectories.
Potential-Based Reward Shaping: To propagate value across extended horizons, the authors employ a potential-based reward shaping update. The immediate reward is augmented by the difference in potential values ( $\Phi$ ), where the potential function is derived from the projected value of the most valuable reachable subgoal. This provides a dense, per-step learning signal that guides the agent toward terminal goals without altering the optimal policy.
Graph Pruning: To ensure the validity of the goal-space planning, a two-stage pruning mechanism (forward and backward breadth-first search) is applied to the goal-transition Directed Acyclic Graph (DAG). This removes disconnected nodes or dead ends that cannot reach the terminal set, ensuring that value iteration operates on a connected subgraph.

Key Contributions

Integration of GSP with DDPG: The paper demonstrates how GSP can be integrated with standard actor-critic RL algorithms to solve long-horizon scheduling problems with terminal constraints.
Sample Efficiency: By utilizing temporally abstract models, the method decouples value propagation from high-frequency dynamics, significantly reducing the number of Bellman backups required to propagate terminal value to early states.
Mitigation of Myopia: The approach explicitly addresses the credit-assignment gap in model-free RL, preventing the agent from sacrificing long-term stability for short-term cost savings.

Results
The proposed method was evaluated on a simulated ASU benchmark over a 72-hour horizon with time-varying electricity prices.

Learning Curves: The "Online GSP" variant (learning models during training) and "Offline GSP" (using pre-trained models) both demonstrated superior sample efficiency compared to standard DDPG. Online GSP approached optimal performance by step 1,000, and Offline GSP converged within 500 steps, whereas standard DDPG struggled to converge effectively within the same timeframe.
Constraint Satisfaction: Standard DDPG policies exhibited myopic behavior, depleting storage early in the horizon and significantly undershooting the terminal storage constraint. In contrast, GSP-based policies maintained elevated storage levels and strategically responded to price signals, consistently satisfying the terminal constraint.
Ablation Study: Removing the state-to-goal projection (Online GSP without projection) resulted in performance degradation, confirming that the fine-grained mapping from states to abstract goals is essential for effective credit assignment.
Computational Overhead: The computational cost of Online GSP was modest, adding approximately 15% to the wall-clock time per episode due to graph reconstruction and value iteration, while Offline GSP added less than 5%.

Significance and Claims
The paper claims that Goal-Space Planning provides an effective mechanism for injecting domain knowledge into model-free RL by constructing goal spaces aligned with key constraint variables. The authors assert that this approach enables agents to acquire interpretable representations of temporal trade-offs, solving the long-horizon credit assignment problem that typically limits the applicability of data-driven methods in process systems. The results suggest that goal-based abstractions are a promising route for tackling other control problems where terminal or path constraints are critical, offering a balance between the flexibility of data-driven methods and the stability requirements of industrial processes.

The Big Picture: The "Flexible Factory" Problem

The Problem: The "Short-Sighted" Student

The Solution: The "Map of Goals"

How It Works in Practice

The Takeaway

More like this